PISA-D Strand C - Sampling Out-of-School Youth

Sampling and Survey Operations –Progress to Date and Lessons Learned

Leyla Mohadjer

Tom Krenzke, Wendy Van de Kerckhove, Lillian Diaz-Hoffmann, Michael Lemay,

Nina Thornton

Westat

PISA D SeminarLondon, UK25 September, 2019

Overview

❯ Sampling objectives

❯ Target population

❯ Sample selection approaches for Strand C (a rare population)

－Representative sampling

－Alternative sampling approaches to reduce cost and burden on countries

❯ Sample selection outcomes

❯ Lessons learned and recommendations for future surveys

2

Strand C - A Pilot Project with Two Main Sampling Objectives

Yield a large enough sample size

3

1

Strand C - A Pilot Project with Two Main Sampling Objectives

Yield a large enough sample size

❯ Test the validity of the

• Questionnaire items

• Assessment items

❯ Allow analyses planned

for Strand C pilot

4

1

Strand C - A Pilot Project with Two Main Sampling Objectives

Yield a large enough sample size

❯ Test the validity of the

• Questionnaire items

• Assessment items

❯ Allow analyses planned

for Strand C pilot

Explore/evaluate various approaches/options

5

1 2

Strand C - A Pilot Project with Two Main Sampling Objectives

Yield a large enough sample size

❯ Test the validity of the

• Questionnaire items

• Assessment items

❯ Allow analyses planned

for Strand C pilot

Explore/evaluate various approaches/options

❯ Arrive at a recommendation

• For identifying and assessing a nationally representative sample of out-of-school 15-year-olds

6

1 2

Strand C - A Pilot Project with Two Main Sampling Objectives

Yield a large enough sample size

❯ Test the validity of the

• Questionnaire items

• Assessment items

❯ Allow analyses planned

for Strand C pilot

Explore/evaluate various approaches/options

❯ Arrive at a recommendation

• For identifying and assessing a nationally representative sample of out-of-school 15-year-olds

7

1 2

Strand A and Strand C Target Populations

8

In-school in grades 7 or above (Strand A)

Division between Strand A and C different across countries

In-school < 7th grade (Strand C)

Out-of-school (Strand C)

Division between out-of-school and in-school <7th grade vastly different across countries and

Factors Impacting Costs and Burden to Countries

❯ Sampling and data collection challenges

• Rare target population

－ About 1 to 3% of the total population

• Individual in-person interviews and assessments

－ In households or other similar locations

• Extensive screening required to locate eligible youth (about 40households to obtain one interview, 64 000 to obtain 1 600 cases)

❯ Critical component of this pilot project

－Arrive at a sampling plan that minimises costs to countries, given the objectives of Strand C

9

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata

10

High concentration

(H)

Low concentration

(L)

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata

11

High concentration

(H)

Low concentration

(L)

Select eligible youth at much higher rate

in stratum H

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata

12

High concentration

(H)

Low concentration

(L)

Probability sample

Probability sample

Select eligible youth at much higher rate

in stratum H

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata

13

High concentration

(H)

Low concentration

(L)

Probability sample

Probability sample

Select eligible youth at much higher rate

in stratum H

Evaluate feasible options for selecting

a nationally representative

sample

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata and two components within each stratum

14

High concentration

(H)

Low concentration

(L)

Probability sample

Non-probability

sample

Probability sample

Non-Probability

sample

Select eligible youth at much higher rate

in stratum H

Evaluate feasible options for selecting

a nationally representative

sample

Stratification and Non-probability Sampling to Reduce Screening Costs

• Divide the country to two major strata and two components within each stratum

15

High concentration

(H)

Low concentration

(L)

Probability sample

Non-probability

sample

Probability sample

Non-Probability

sample

Select eligible youth at much higher rate

in stratum H

Evaluate feasible options for selecting

a nationally representative

sample

Yield a large enough sample size at a

substantially lower cost

Example - Guatemala : High/Low Concentration areas defined as Rural/Urban areas

16

Sampling Approaches Used in Strand C

1. Representative Probability

2. Link-tracing through householder referrals or recruiting

3. Limited Representative

a) School frame approach for out-of-school youth

b) School frame approach for <7th grade

c) Location sampling

17

Sample Design - Cluster Sampling to Reduce Travel Costs

PSU

DU

Two stage sample within High and Low density strata• Stage 1

• Primary Sampling Units (PSUs)• Stage 2

• Dwelling Units (DUs)

Several countries conducted mini-

censuses in the PSUs

Stratum High/Low

Main Study – Screening to locate eligible youth

19

0 5,000 10,000 15,000 20,000 25,000 30,000 35,000

Guatemala

Honduras (rural only)

Panama (indigenous andrural only)

Paraguay

Senegal

Representative Probability Sample - Number of dwelling units sampled in each country

High density Low density

Main Study – Representative Probability Samples

20

0 200 400 600 800 1000 1200 1400 1600 1800

Guatemala

Honduras (rural only)

Panama

Paraguay

Senegal

Number of Youth Interviews and Assessments

Low density Youth interview and assessment Low density Youth interview only

High density Youth interview and assessment High density Youth interview only

Number of Completed Cases by Country

Core Total Target Sample Size = 1 600 (Minimum Probability Sample = 1 200)

21

0

200

400

600

800

1000

1200

1400

1600

1800

2000

2200

2400

Guatemala Honduras Panama Paraguay Senegal

Num

be

rof com

ple

tes

LR

LTHH

Prob L

Prob H

Prob H = representative probability sample high density strata; Prob L = representative probability low

density strata; LTHH = link-tracing through households; LR = limited representative sample

Lessons Learned and Recommendations

22

Lessons Learned and Recommendations

❯ National centre relationship with the National Statistical Institute is

crucial for Strand C success

❯ Critical to have access to

• Reliable population counts and maps

• Project staff with experience in household sampling and data collection

• Experienced and adequate number of interviewers

23

Lessons Learned and Recommendations (2)

❯ Countries budgets and administrative processes are not conducive to

household survey sampling and operations needs

❯ All aspects of sampling and survey operations take longer than expected

❯ Quality monitoring and quality control are challenging

❯ Logistical challenges in accessing the most disadvantaged areas in

developing countries are significant

24

Summary and Conclusions

❯ Countries worked diligently throughout the Field Trial (FT) and Main

Study (MS) process despite of numerous challenges (budget, access to

needed staff and information, etc.)

❯ Pilot study sampling and survey operations goals were met

25

Thank You leylamoahdjer@Westat.com

Photos are for illustrative purposes only. All persons depicted, unless otherwise stated, are models. 26