RTI International is a trade name of Research Triangle Institute

Sampling U.S. Schools – Frame, Sampling & Field Issues

International Conference on Establishment Surveys

June 21, 2007

Cynthia Augustine, Karol Krotki, Deborah Herget

2

Overview

Frame Issues Coverage, completeness

Accuracy

Sampling Topics Stratification

Multistage Sampling

Field Processes Response Rates

Strategies for Increasing Response Rates

Conclusions

3

2006 PISA and PIRLS

National Center for Education Statistics (NCES)

PISA (Program for International Student Assessment)

15 Year Old Students

Multiple Subject Areas

PIRLS (Progress in International Reading Literacy Study)

4th Grade Students

Literacy

4

Frame Issues

US Department of Education’s of Common Core of Data (CCD) and Private School Survey (PSS) for 2003-2004

Supplementary Information School sizes

Grade sizes

Federal school lunch program (proxy for Poverty). Private Schools assumed low poverty

5

Frame Issues

Accuracy In 2 field tests, all

sampled schools were located

Good Coverage

2000-2005 Census:

PISA Frame

4,158,362 4,156,069

Number of 15 Year Olds

Combined Frame

2006 QED

Students 57,879,426 50,910,308

Schools 140,998 115,512

6

Frame Issues

Out of date frames are a challenge in establishment surveys Also a problem with PISA / PIRLS: 2003-2004 datasets for

Winter 2006 data collection

Methods for updating frames Request updated school lists from Districts

Updating on the fly – Half Open Interval (HOI) methods

Check Actual v. Frame to identify new schools and potentially re-sample

Sample Districts rather than schools

7

Sample Design

PISA used implicit stratification by sorting the file by Public/Private, Region, Locale, High Minority Percent, and Grade Span

N=236 Original Schools

PIRLS used 10 Self-Representing Metropolitan Statistical Areas (MSAs) and 44 Primary Sampling Units (PSUs).

N=252 Original Schools

8

Sampling Issues

PSU Creation

Manually created within states following Metropolitan Statistical Areas (MSAs) and counties

Total Number of students in PSU

Multistage Cluster Sampling (PIRLS Only)

44 PSUs selected with PPS (Students in PSU)

4 Schools selected PPS (Students in School)

Classrooms selected within schools using KeyQuest, an international sampling software

9

Sampling Issues

Measures of Size PISA: Estimated number of 15 year olds PIRLS: Number of 4th grade students

Oversampling High Poverty (50% Eligible for Federal Lunch Program) Private Schools

10

Sampling Issues

Probability Proportional to Size sampling will make school-level weights inaccurate without multiplying by the Measure of Size.

Cluster sampling reduces travel costs and contacts with multiple school districts

11

Field Processes

Low & Falling Response/Cooperation Rates among Schools

Preliminary Response Rate: PISA: 77% PIRLS: 84%

Strategies for Increasing Cooperation

Financial Incentives for students

Out-of-School Administration

PISA: “The Global Context of Scientific Literacy” Conference

PIRLS: Winter Olympics

12

Conclusions

CCD and PSS provide good coverage, accuracy and supplementary information and are free.

Cluster sampling reduces travel costs and number of school districts contacted.

Incentives are important for increasing cooperation rates.

13

Questions or Comments

Cynthia AugustineRTI International

3040 Cornwallis RoadResearch Triangle Park, NC 27709

919.541.6154caugustine@rti.org

14

References

U.S. Census Bureau

Hamann, Thomas A. “Evaluating the coverage of the U.S. National Center for Eduation Statistics’ Public / Elementary School Frame.” American Statistical Association 1999.

Quality Education Data, Inc. http://www.qeddata.com/Catalogcounts/DG_FINAL_2006.pdf