Welcome to Fantasyland: Comparing Approaches to Land Area Measurement in Household SurveysSydney GourlaySurvey Specialist

Living Standards Measurement Study TeamDevelopment Research Group, The World Bank

sgourlay@worldbank.org

2015 World Bank Conference on Land and Poverty

Motivation & Key Questions• Land area is critical in:

– Measuring yields– Land distribution– Titling schemes– Anything agriculture…

• Gaps in literature:– How do measurement

methodologies stack-up?• Is GPS as accurate as the gold-

standard?– What characteristics influence

measurement accuracy?– How does the measurement

methodology employed influence analytical outcomes?

Measuring Land Area: Methodological OptionsFarmer self-reported

estimate

PROS- Inexpensive

- Less missingness

CONS- Subjective

- Complicated by traditional units

-Potential ulterior motives

Compass and rope (aka traversing)

PROS-Traditional gold

standard for accuracy- Eliminates subjectivity

CONS- Time/labor intensive

(leading to higher costs)

- Requires travel to plot

GPS

PROS- Significantly quicker than traversing with

advantages of objective

measurement

CONS- Questions of

accuracy on small plots (?)

- Requires travel to plot

Remote Sensing (?)

PROS - Potential to

eliminate plot visits

CONS- Resolution limitations

- Feasibility of boundary

identification

LSMS Methodological Validation Program (MVP)• UK Aid-funded project: Improving Measurement of Agricultural

Productivity through Methodological Validation and Research

• To date, 3 highly-supervised methodological studies on land area measurement:

• Ethiopia• Tanzania• Nigeria

Methodological Validation ExperimentsLand and Soil Experimental

Research (LASER)

Improving the Measurement of

Cassava Productivity

Area Measurement

Validation

Location Ethiopia Zanzibar, Tanzania Nigeria

Components:Land Area;

Soil Testing; Crop Cutting

Land Area; Production Diaries;

Crop CuttingLand Area

Partnerships:

World Agroforestry

Centre (ICRAF), Central

Statistical Agency

Ministry of Agriculture and

Natural Resources, Office of the Chief

Government Statistician

National Bureau of Statistics

Fieldwork Complete

March 2014 July 2014 May 2013

1798Ethiopia

1945Tanzania

494Nigeria

4237Total

Plots with GPS and Compass & Rope

These studies included farmer self-reported estimate, GPS, and compass & rope measurement

Scope of (Preliminary) Analysis• Comparison of competing measurements:

– Bias: Difference between two measurements (in acres)• For example: GPS - CR

– Absolute Value of Bias• For example: |GPS – CR|

– Relative Bias: Difference between two measurements (in terms of %)• For example: (GPS- CR)/CR * 100%

– Absolute Value of Relative Bias• Analysis of systematic measurement error between measurements via OLS regression• Determinants of “High Bias” plots

Comparison of Measurements: Subjective vs Objective (1)SR estimates are sensitive to respondent characteristics, including the tendency to round off numbers:

0

10

20

30

Perc

en

t

0 1 2 3Acres

SR GPS

*Limited to plots <=3 acres

Plot Area Distribution

LSMS Malawi 2010/11

Comparison of Measurements: Subjective vs Objective (2)SR vs CRAcres

Level (CR)Ethiopia Tanzania Nigeria

N SR CR BiasMean Bias /

Mean CR N SR CR BiasMean Bias /

Mean CR N SR CR BiasMean Bias /

Mean CR1 (< 0.05 acres) 352 0.09 0.02 0.07 307% 44 0.32 0.04 0.28 661% - - - - -2 (< 0.15 acres) 392 0.27 0.09 0.18 188% 622 0.41 0.11 0.31 288% 21 0.15 0.11 0.03 30%3 (< 0.35 acres) 351 0.40 0.23 0.17 72% 816 0.62 0.23 0.39 173% 73 0.39 0.25 0.14 55%4 (< 0.75 acres) 316 0.66 0.51 0.15 29% 323 0.98 0.49 0.49 100% 129 0.79 0.53 0.26 50%5 (< 1.25 acres) 179 0.95 0.97 -0.02 -2% 63 1.53 0.92 0.61 66% 108 1.31 0.99 0.32 33%6 (>= 1.25 acres) 99 1.42 1.90 -0.47 -25% 20 2.05 1.81 0.24 13% 154 2.57 2.87 -0.30 -10%

Total 1689 0.47 0.38 0.09 23% 1888 0.65 0.27 0.38 143% 486 1.38 1.31 0.07 5%

Pooled data reveals self-reported estimates are over-estimated by 51% on average (over CR measurement)

Comparison of Measurements: Subjective vs Objective (3)0

2

4

6

0 2 4 6GPS

Ethiopia

0

2

4

6

CR

0 2 4 6SR

0

1

2

3

0 1 2 3 4GPS

Tanzania

0

1

2

3

CR

0 2 4 6 8SR

05101520

0 5 10 15 20GPS

Nigeria

05101520

CR

0 5 10 15 20SR

Ethiopia

Tanzania

Nigeria

Comparison of Measurements: Objective vs Objective (1)GPS vs CRMeans; Acres

Level (CR)

Ethiopia Tanzania Nigeria

GPS CR Bias

Mean Bias /

Mean CR GPS CR Bias

Mean Bias /

Mean CR GPS CR Bias

Mean Bias /

Mean CR1 (< 0.05 acres) 0.02 0.02 0.00 0% 0.04 0.04 0.00 -3% - - - -2 (< 0.15 acres) 0.10 0.09 0.00 2% 0.11 0.11 0.00 2% 0.11 0.11 -0.01 -7%3 (< 0.35 acres) 0.24 0.24 0.01 3% 0.23 0.23 0.01 2% 0.24 0.25 -0.01 -4%4 (< 0.75 acres) 0.52 0.51 0.01 2% 0.51 0.49 0.02 4% 0.52 0.53 -0.01 -2%5 (< 1.25 acres) 0.98 0.96 0.02 2% 0.94 0.92 0.02 2% 0.97 0.99 -0.02 -2%6 (>= 1.25 acres) 1.91 1.89 0.02 1% 1.91 1.81 0.09 5% 2.87 2.87 0.00 0%

Total 0.38 0.38 0.01 2% 0.28 0.27 0.01 3% 1.30 1.31 -0.01 -1%

Pooled data reveals GPS measurements are over-stated by 1% on average (over CR measurement)

Comparison of Measurements: Objective vs Objective (2)0

2

4

6

0 2 4 6GPS

Ethiopia

0

2

4

6

CR

0 2 4 6SR

0

1

2

3

0 1 2 3 4GPS

Tanzania

0

1

2

3

CR

0 2 4 6 8SR

05101520

0 5 10 15 20GPS

Nigeria

05101520

CR

0 5 10 15 20SR

Correlation between GPS and CR measurements: 0.997 (pooled data)

Comparison of Measurements: Objective vs Objective (3)Despite high correlation of GPS and CR measurements, evidence of “high bias” or problem plots exists:

High bias plots are generally:• smaller than average, • have a higher closing error (suggesting bias is at least partially attributable to CR), and • have a more complex shape (proxied by perimeter:area ratio).

Comparison of Measurements: Objective vs Objective (4)We attempt to explain the discrepancy between GPS and CR by:

Yi=Li+Ci+Si+SATi+Ti+Wi+ei

- Y is one of the four measures of bias, - L is the measure of the plot taken using CR, - C is the closing error of the CR measure, - S is a vector of proxies for the shape of the plot, - SAT is the number of satellites the GPS device had acquired, - T is a vector of dummy variables related to tree canopy cover, - W is a vector of dummy variables related to weather conditions, and - e is a random error.

Comparison of Measurements: Objective vs Objective (5)The small difference is difficult to capture, but:• Heavy tree cover & cloudy conditions slightly increase absolute value of percent deviation between measurements (in Ethiopia and Tanzania)• Perimeter : Area ratio is statistically significant in all countries, suggesting that in plots with more complex shapes the absolute value of percent bias is higher.• Plot area is the most significant factor…

• Plot area is the most significant factor…-.0

3-.

02

-.0

10

.01

GP

S-C

R (

ac

res

)

0 1 2 3P lot Size (acres, CR)

0.0

2.0

4.0

6.0

8.1

|GP

S-C

R|

(ac

res

)

0 1 2 3Plot Size (acres, CR)

-20

24

Re

lati

ve B

ias

(%)

0 1 2 3Plot S ize (acres, CR)

45

67

89

Ab

solu

te V

alu

e o

f R

ela

tive

Bia

s (

%)

0 1 2 3Plot Size (acres, C R)

Bias (acres)Ethiopia

| Bias | (acres) Relative Bias (%) | Relative Bias | (%)

0.0

5.1

.15

GP

S-C

R (

ac

res

)

0 1 2 3P lot Size (acres, CR)

0.0

5.1

.15

.2|G

PS

-CR

| (a

cre

s)

0 1 2 3Plot Size (acres, CR)

24

68

10

Re

lati

ve B

ias

(%)

0 1 2 3Plot S ize (acres, CR)

02

46

8A

bso

lute

Va

lue

of

Re

lativ

e B

ias

(%

)

0 1 2 3Plot Size (acres, C R)

Tanzania

-.06

-.05

-.04

-.03

-.02

-.01

GP

S-C

R (a

cres

)

0 2 4 6 8 10P lot Size (acres, CR)

0.1

.2.3

|GP

S-C

R| (

acr

es)

0 2 4 6 8 10Plot Size (acres, CR)

-10

12

3R

ela

tive

Bia

s (%

)

0 2 4 6 8 10Plot S ize (acres, CR)

-.1

-.05

0.0

5A

bso

lute

Va

lue

of R

elat

ive

Bias

(%)

0 2 4 6 8 10Plot Size (acres, C R)

Nigeria

Bias (acres) | Bias (acres)| Relative Bias (%) | Relative Bias (%) |

Comparison of Measurements: Objective vs Objective (7)

0

20

40

60

80

100

Ethiopia Tanzania Total

GPS CRPlot Size Level (CR) & MinutesAverage Measurement Duration

• Ethiopia:– GPS = 13.7 minutes– CR = 56.8 minutes

• Tanzania:– GPS = 7.4 minutes– CR = 29.3 minutes

Final Thoughts• Lower limit to GPS use not yet clearly defined(But lower than most literature alludes to)• GPS + SR

– When GPS measurements are missing, impute them using the self-reported area estimates a la Kilic et al:• Kilic, Zezza, Carletto, and Savastano. “Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements.” World Bank Policy Research Working Paper No. 6490.

Welcome to Fantasyland: Comparing Approaches to Land Area Measurement in Household SurveysSydney GourlaySurvey Specialist

Living Standards Measurement Study TeamDevelopment Research Group, The World Bank

sgourlay@worldbank.org

2015 World Bank Conference on Land and Poverty