1

Ministry of Finance and Ministry of Samurdhi

Welfare Benefits Board

The Targeting Formula: Analysis Using Pilot Data

Welfare Workshop

Colombo, 16-17 November 2003

2

Developing criteria for identifying welfare beneficiaries

The idea of proxy means test formula (PMTF): to find a weighted combination of “proxy” variables/indicators that together identify or predict whether a household is poor or not

The PMTF assigns a “score” to every household, based on information collected from the household for all variables that are included in the formula

Choice of proxy variables must balance between 3 criteria: able to identify the poor with some accuracy; easily observable and measurable; cannot be manipulated easily by household

PMTF developed by regression analysis using household surveys Analysis is used to identify appropriate set of variables and weights Alternative models from Sri Lanka Integrated Survey (SLIS, 1999-2001),

and Consumer Finance Socio-Economic Survey (1996-97)

3

The Proxy Means Test Formula using SLIS

Notes:1) All scores are derived from regressions of (log of) per capita consumption expenditure on a set of variables2) The score for each variable is its coefficient in the regression, multiplied by 100, and rounded to the nearest integer3) Lower the score, poorer the household is considered to be3) The aggregate score for each household is calculated as constant +/- the weight on each variable. For each 0-1 variable, multiply the score by 1 if true for household, by 0 if not true. For each continuous variable (denoted by *), multiply the score by the value of the variable for the household)

Variables Weights

Community characteristics Public/Private bank in community 8 Divisional Secretariat in community 9

Household assets Car/van 40 Cooker (kerosene/gas/electric) 17

Bicycle/Tricycle 4 Fan 11 Refrigerator 12 Motorcycle/Scooter 8 Radio/CD/Cassette player 4 Sewing Machine 7 Tractor 15 TV/Video player 8

Land and livestock Cultivable land owned by household : 1<Acres<=2 7

2<Acres<=4 8 Acres>4 16

Livestock (any) 8 Household head

Not a female who is widowed/separated/divorced 5 Age: 70-79 -5 80 and above -13 Education: Passed OL or Grade 11 7

Passed AL/GAQ/GSQ 10 Has Degree/PG/Diploma 16

Work: Salaried employment or in business 5 Household demographics

Household size: 3-4 members -23 5-6 members -39 7-8 members -52 >8 members -59

All children age 5-16 attend school 6 Housing characteristics

Dwelling owned by hhold 4 Fuel for cooking: Gas/electricity 13 Toilet: Private and flush type 16 No. of Rooms (excl. kitchen/bath) per hhold member* 16 Walls: Not cabook/mud/plank/cadjan 6

Constant 707

4

Validating the formula through a pilot

Pilot targeting exercise conducted over the last few months

Information collected on variables in the 2 formulas through application forms in selected GN divisions

Analysis of data collected to test how the predictions from the formula measure up against real data from an application process

The Pilot Applicants

District Name No. of GN Divisions

No. of Households

COLOMBO 3 1,739 KALUTARA 5 845 KANDY 4 862 NUWARA - ELIYA 14 4,104 GALLE 23 2,652 MATARA 6 1,680 HAMBANTOTA 21 2,973 VAVUNIYA 6 1,813 TRINCOMALEE 17 5,063 BADULLA 4 695 MONARAGALA 5 2,282 RATNAPURA 6 1,995 Total 114 26,703

5

Key questions analyzed using pilot data

What was the rate of response to the pilot application, and how well has the application form worked?

How does the coverage (% of population identified as beneficiaries) of the formula for different cutoff scores measure up to what was simulated from survey data?

What are the implications of coverage figures for the choice of cutoff points for eligibility ?

What can we say about coverage of groups that are likely to be especially vulnerable ?

How do the results compare across different regions of the country?

What implications can be drawn for field work or data collection efforts ?

6

Response to pilot targeting exercise and implications

58% (in terms of population) response rate for all pilot areas: 55% in non-NE areas, 69% in the 2 NE districts

What we expect: people not interested in program would not fill up the form Concern: did everyone who would have been interested fill up the form ?

Special concern: those who have been “missed” by the program so far, did they fill the form?

What about those who filled, but information is incomplete? Because of non-response, all population coverage estimates will have to be calculated

assuming that all who did not apply would not have qualified for the program

* denotes estimates using average household size from census information (for non-Northeast districts), since population figures for Northeast districts are not available

Census No. of Applicants Completed applications

Population Households Population Households Population Households 199,151* 48,501 115,580 26,703 107,418 24,660 All

Districts (58.0%) (55.1%) (53.9%) (50.8%)

156,583 38,134 86,309 19,827 80,154 18,299 Non-NE Districts (55.1%) (52.0%) (51.2%) (48.0%)

42,568* 10,367 29,271 6,876 27,264 6,361 NE Districts (68.8%) (66.3%) (64.0%) (61.4%)

7

How serious is the problem of incomplete information for the purpose of analysis?

Age of household head is the most frequently missing variable No evidence that missing information differentially affects certain TYPES household,

which would have caused a bias in results due to missing variables

There are some variables that were imperfectly measured: cultivable land, type of toilet, homeless

Variables # households

missing Assets 178 Housing 100 Rooms per member 419 Single female household head 630 Head’s age 950 Head's education 336 Head’s main occupation 790 Cultivable land 320 Household size 7

# households Any missing variable 2166

No missing variable 24660 Homeless households 354

8

Cumulative distribution of pilot scores from SLIS model:Comparison with results from SLIS sample

score

Pilot SLIS SLIS_actual

600 650 700 750 800 850 900 950 1000

0

.2

.4

.6

.8

1

The pilot sample is poorer than the SLIS national sample

Expected since pilot sample contains only those who are interested in the program

SLIS_actual is log per capita expenditure times 100, SLIS refers to predicted score from SLIS data, and Pilot score is calculated using the SLIS model on all pilot sample cases for which the model score could be calculated

SLIS Actual

SLIS predictedPilot sample

9

Applying SLIS model to pilot: program coverage

% covered in pilot sample for every cutoff point is higher than that predicted from SLIS –since pilot applicants are poorer

Important: % covered among total population in pilot districts for every cutoff point is close to that predicted from SLIS The difference between predicted and pilot population coverage becomes higher as

cutoff point is raised This difference also suggests that all the poor households probably did not apply for

the pilot program

Coverage (%) : Non NE Coverage (%) : Entire

pilot sample Percentile of

actual per capita

consumption (SLIS)

Cutoff scores from

SLIS model

Coverage (%) predicted from SLIS Pilot

Sample Pilot

Population Pilot

Sample Pilot

Population

20th 695 12.0 20.4 11.2 20.3 11.8 25th 703 19.6 33.0 18.1 32.6 18.9

30th 709 26.3 42.8 23.6 42.5 24.6 35th 715 33.7 52.8 29.1 52.7 30.5 40th 721 41.9 62.6 34.5 62.4 36.2

10

Cutoff points and coverage

Coverage is higher for the two NE districts: a higher proportion of applicants in the total population brings the coverage closer to what is predicted

Choice of a cutoff point: recognize that the proportion of applicants in the population is expected to rise in the actual program, which will raise the % of population covered for a given cutoff E.g. with cutoff at a score of 709, pilot population coverage of 24.6% is

lower than predicted coverage of 26.3% But coverage will increase if strong efforts are made to have all the poor

apply to the program Thus 24.6% is lower bound of expected coverage; actual coverage should

be expected to be 2-3 percentage points higher

Last but not the least: recall the inherent tradeoff in choice of cutoff due to the formula being unable to identify the poor “perfectly” Higher the cutoff, higher the rate of “leakage” to non-poor Lower the cutoff, lower the rate of coverage among the poor

11

Coverage among specially “vulnerable” groups on applying the formula: high in pilot sample

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households with DISABLED heads

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households with SINGLE FEMALE heads

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households with AGE 70+ heads

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households with cultivable land<1 acre

SLIS actual SLIS predicted Pilot sample

Cutoff scores

Coverag

e

12

Coverage among other vulnerable groups also high in pilot sample…..

0

0.2

0.4

0.6

0.8

20th 25th 30th 35th 40th

Households with 7 or more members

0

0.2

0.4

0.6

0.8

20th 25th 30th 35th 40th

Number of CHILDREN (age 0-16)

As is coverage among large households & children…..

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households who do not OWN house

0

0.2

0.4

0.6

20th 25th 30th 35th 40th

Households who do not OWN livestock

SLIS actual SLIS predicted Pilot sampleCutoff scores

Coverag

e

13

Coverage of the poor in specially vulnerable groups

Results from household survey (SLIS) show little reason for concern Undercoverage (proportion of poor “missed” by formula) is less for all

vulnerable groups than among the poor in general, for all cutoff points

Undercoverage Rates using SLIS formula Population of Poor Households with

Cutoff Overall

population of poor

disabled head

female single head

land owned<1 acre

not own house

age 70+ head

household size 7+

20th 0.64 0.62 0.56 0.62 0.52 0.54 0.40 25th 0.52 0.51 0.50 0.52 0.43 0.42 0.28 30th 0.43 0.40 0.41 0.41 0.35 0.33 0.20 35th 0.37 0.32 0.35 0.34 0.28 0.32 0.17

40th 0.28 0.26 0.27 0.26 0.21 0.25 0.12

Note: Calculations based on SLIS sample – representative for Sri Lanka, excluding the North & East

14

Coverage of vulnerable households who are in the poorest 30% of the population

When cutoff = 709 (30th percentile of actual consumption)(Results using SLIS data)

Disabled head of household Single female head of household

Cultivable land owned < 1 acre Household head's age > 70 years

Covered w hencutoff of 709 (30thpctile)

Not covered

15

Comparison between North & East districts and the rest

Significantly higher response rate to pilot application in N & E districts Leads to higher % of population covered (for any cutoff) in NE districts

than in non-NE districts

Most household characteristics in N&E pilot districts – defined by the variables in the formula – are not significantly different from those in the other pilot districts 3.1% of households in NE districts have a disabled head, compared to 4.8%

of households in non-NE; 14.0% of households in NE have a single female head, compared to 12.6% of those in non-NE

Coverage (%) in pilot sample

Coverage (%) in population in pilot

districts

Percentile of actual per capita

consumption (SLIS)

Cutoff score from SLIS

model

Coverage (%) predicted from SLIS

NE non NE NE non NE 20th 695 12.0 19.9 20.4 13.8 11.2 25th 703 19.6 31.2 33.0 22.1 18.1 30th 709 26.3 40.9 42.9 28.9 23.6 35th 715 33.7 51.3 53.0 36.4 29.1 40th 721 41.9 61.1 62.8 43.0 34.5

16

Comparison with groups of existing program beneficiaries

Distribution of Samurdhi beneficiaries in pilot somewhat correlated with SLIS formula scores in N&E; little correlation between the two in non-N&E districts

If cutoff was set at the 30th percentile, to benefit about 25% of pilot population

63% of beneficiaries are current Samurdhi recipients, 6% benefit from other welfare programs (but not Samurdhi), and 31% currently receive no benefits

Eligible households include: 47% of current Samurdhi recipients, 45% of those who receive other welfare benefits only, and 35% of those with no welfare benefits

North & East districts

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10

Deciles by pilot scores using SLIS formula

O ther pilot districts

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10

No welfare Samurdhi Other Welfare program

17

The homeless households

All homeless households should be in the program automatically The formula is not applicable for them; thus all results so far have been derived after

taking the homeless out of the sample

Proportion of homeless in the pilot sample Does not change between NE and non-NE districts Significantly higher in rural areas Q: Have all the homeless been able to apply?

Average household size much lower for homeless households 24% of homeless households comprise of a single member, compared to 1% of those

living in homes 5% of homeless households comprise of a 6 or more members, compared to 37% of

those living in homes

House-hold size

% of Non homeless

% of homeless

1 1.0 24.3 2 5.0 23.4 3 13.2 26.3 4 21.1 11.1 5 22.8 9.7

6+ 36.9 5.2

Total 100 100

% of homeless households

# homeless households All Rural Urban

ALL districts 354 1.33 1.42 0.41 Non – N&E 266 1.43 1.47 0.42

N&E 88 1.28 1.30 0.00

18

Lessons for field work: 1) formula is meant to replace subjective judgment

A single variable in most cases is not a good predictor for poverty With cutoff set at the 30th percentile (score=709), 26% of selected

households has TV/VCR, 41% has electric light, 50% has brick/cement wall Thus critical to collect all information, and not form pre-judgments about a

household’s poverty based on inspection – let the formula do its job !

0

10

20

30

40

50

60

70

20th 25th 30th 35th 40th

Cutoff percentiles

% V

alu

es

TV/VCR Fridge okay quality w all

okay quality f loor electricity for light gas/electricity for cooking

19

Lessons for field work: 2) all variables are necessary, but some are especially critical to measure accurately

Ten most “sensitive” variables for identifying the poor using SLIS formula Sensitivity depends on the the variable’s weight and its distribution Coverage is most affected by household size, rooms per member, and whether

household head is female and single

0

5

10

15

20

25

hhsiz

e5-6

room

per m

embe

r

hhsiz

e7-8

no fe

mal sin

gle wall

hhsiz

e3-4

dwell

own

tv/vid

eo

all ki

ds a

ttend

hhsiz

e9+

Cutoff at 30th percentile of consn (score=709) Cutoff at 25th percentile of consn (score=703)

Change in coverage (absolute value) when one variable is dropped from score

0

5

10

15

20

25

hhsiz

e5-6

room

per m

embe

r

hhsiz

e7-8

no fe

mal

single wall

hhsiz

e3-4

dwell

own

tv/vid

eo

all ki

ds a

ttend

hhsiz

e9+

20

Implications for a broader safety net system

There remain certain categories of poor and vulnerable households that are less likely to be picked up by the formula Poor households who are small in size ……who have to support disabled members (not necessarily the head of the

household) ……who have suffered a recent reversal of fortune (e.g. death/disability of a

primary earning member, loss of crops) ……affected by civil strife/violence/dislocation, unless the effect is directly in

terms of loss of assets or disability to household head ……affected by illness of head/other household members

Since the above are relatively rarely occurring events or shocks, the formula cannot incorporate these as variables that explain poverty for the general population

Such cases underscore the need for a broader safety net system, incorporating programs that address specific needs and vulnerabilities