Embed Size (px)
Citation preview
1
Meeting Data Collection Meeting Data Collection Challenges of the FutureChallenges of the Future
James GriffithJames Griffith
Ted Socha Ted Socha
Thomas WekoThomas Weko
Postsecondary Studies DivisionPostsecondary Studies Division
National Center for Education National Center for Education StatisticsStatistics
2
Purpose of the Purpose of the PresentationPresentation
To show how our “technological innovations” have responded to challenges facing sample survey
and longitudinal study data collections.
3
OverviewOverview
• Provide overview of postsecondary sample survey and longitudinal studies as prelude to …
• Major challenge – Encourage participation of institutions and students
• Technological innovations to meet this challenge
• Future challenges, concluding remarks
4
Postsecondary Longitudinal and Postsecondary Longitudinal and Sample Survey StudiesSample Survey Studies
• National Postsecondary Student Aid Study (NPSAS)
• Beginning Postsecondary Students Longitudinal Study (BPS)
• Baccalaureate and Beyond Longitudinal Study (B&B)
• National Study of Postsecondary Faculty (NSOPF)
5
Primary PurposesPrimary PurposesNPSAS• NPSAS:08, includes 125,000 undergraduate and 13,000
graduate/first-professional students enrolled in 1,963 institutions
• Describes how students and their families pay for postsecondary education and the role of federal student aid
• Describes undergraduate and graduate student populations, including topics such as civic participation, community service, educational aspirations, life goals, disabilities
• Provides rich database for postsecondary research and policy analysis, data gathered on over 1,100 variables
• Responds to emerging federal policy interests, such as debt level, use of private loans, training in STEM majors, and state-level financial aid issues
6
Primary PurposesPrimary PurposesBPS• Includes about 18,500 first-time beginning students
identified in NPSAS, followed 2 years and 5 years later• Provides data on student “flow” in and out of
postsecondary education, such as persistence, transfer behavior, attainment
• Gathers data on initial work experiences of those who obtain certificates or 2-year degrees
B&B• Includes about 23,500 bachelor degree completers
identified in NPSAS, followed 1 year, 5 years, and 10 years later
• Provides important information on graduate education, work, and career through retrospective information on:-- postsecondary experiences -- paths taken to the degree award, time to degree completion
• Focuses on “teacher pipeline,” interest in teaching, preparation, and job search
7
Chronology of PLSSS StudiesChronology of PLSSS Studies
National Postsecondary Student Aid Study (NPSAS)1987, 1990, 1993, 1996, 2000, 2004, 2008
1986-871989-90
BPS1992-93
B&B1995-96
BPS1999-2000
B&B2007-2008
B&B
1992
1994
1994
1997
2003
1998
2001
2001 2009
2012
2003-2004BPS
2006
2009
BPS = Beginning Postsecondary Students Longitudinal Study B&B = Baccalaureate and Beyond Longitudinal Study
8
Major Data CollectionsMajor Data Collections
NPSAS occurs in staged sampling periods requiring data collection at several levels:
• Institution data collection
• Student interview data
• Other administrative data
9
Data Collections by StudyData Collections by StudyNPSAS BPS B&B
Institution-Level Data
Institution characteristics (IPEDS data) × × ×
Student records (institutional and state aid) ×
Transcripts × ×
Student Interview Data × × ×
Other Administrative Data
Central Processing System (e.g., FAFSA) × × ×
NSLDS (Pell and loan files) × × ×
National Student Clearinghouse (enrollment) × × ×
ACT / SAT / Praxis × ×
10
The ChallengeThe ChallengeDeclining Institution and Student Response Rates
Survey
Study
Student-level CADE
response rate
Student interview
response rate
Data Collection
Period
Unweighted
Weighted
Unweighted
Weighted
NPSAS:96
93.0
93.0
Not available – 2
phases
76.0
August 1996 -- February
1997 (CADE first)
NPSAS:00 92.0 97.0 70.0 72.0 March 2000 – February 2001
(CADE first)
NPSAS:04 88.0 91.7 62.6 70.6 February 2004 – September
2004 (CADE and CATI
simultaneously)
11
• Declining participation of institutions and students
• Missing data at the case-level and item-level
• Complicated by …
-- Diversity in access and use of technology across institutions (about 6,700) and students (about 19 million)
-- Data privacy/security concerns
Government = Misuse of study data
Institutions = Risks of releasing student information
Families = Privacy, identity theft
-- Tight schedule for data collection, processing, and delivery – from lists of students gathered in May to data delivery in January
-- Students as transient population
GOAL – To maximize the usefulness and quality of the data vis-à-vis the challenge and complicating factors …
12
Institutional Data CollectionInstitutional Data CollectionChallenge: Maximize institution participation by
acquiring enrollment lists for sampled institutions.
Innovations:
• Contacted early (fall contact for spring collection)
• Established “Study Coordinator” (IRP as initial commitment)
• Provided Help Desk (10-12 staff, Mon-Fri work hours)
• Provided easy access through Website (information, secure login, CADE)
• Ensured confidentiality / security (FERPA documents, IRBs)
• Offered multiple options for participation (secure website, encrypted fax, hard copy)
13
Institutional Data CollectionInstitutional Data Collection
Challenge: Minimize institution nonresponse.
Innovations:
• Developed real-time monitoring of characteristics of institutions (IMS)
• Performed bias analysis for < 85% participation overall and/or within strata, NCES standard
• Made weight adjustment to reduce bias at the institution/unit level
14
Real-time Monitoring SystemReal-time Monitoring SystemWe can see our response statistics in real-time.
We then target “under-responding” institutions for participation in near real-time.
15
Institutional Data CollectionInstitutional Data Collection
Challenge: Maximize student record collection.
Innovations:
• Provided accessible Web-based instrument (CADE)
• Provided Help Desk (Mon-Fri during normal work hours)
• Negotiated reimbursement to “incentivize” participation
16
Example – Example – Web CADE Web CADE Student Student Record Record AbstractionAbstraction
17
CADE abstraction method
Institutions providing CADE Total students1
Number Percent2 Number Percent2
Total 1,300 100.0 103,620 100.0
Abstraction method
Web-CADE 860 65.9 48,860 47.2
Data-CADE 280 21.1 33,210 32.0
Field-CADE 170 13.1 21,550 20.8
Student Record Abstraction Student Record Abstraction Method – NPSAS:04Method – NPSAS:04
1 The total represents the number of students sampled from institutions that completed computer-assisted data entry (CADE) and may include students who were classified as study nonrespondents.2 Percentage of total number of eligible institutions/students.NOTE: Detail may not sum to totals because of rounding.SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).
Web CADE preferred choice
18
Institutional Data CollectionInstitutional Data Collection
ChallengeChallenge: : Maximize student transcript Maximize student transcript collection.collection.
InnovationsInnovations::
• Addressed confidentiality/privacy concerns Addressed confidentiality/privacy concerns through secure fax, encrypted email, FTP through secure fax, encrypted email, FTP transfer transfer
• Negotiated reimbursement to “incentivize” participation
19
Example – Example – Web Web Transcript Transcript CollectionCollection
20
Student Interview Data CollectionStudent Interview Data Collection
Challenge: Maximize student interview collection.
Innovations:
• Redefined completed case
• Employed multiple tracing vendors, e.g., CPS, NCOA, Telematch, Equifax, Experian,TransUnion
• Offered response mode options fitting to the population (self-administered via Web, telephone, face-to-face)
• Provided accessible Web-based instrument
• “Incentivized” for early response
• Provided Help Desk (7 days a week)
21
Cases Have Multiple Data Sources … Cases Have Multiple Data Sources … Primary sources
Institution records (CADE) 95%
Student interviews 70%
Federal aid applications (CPS) 60%
Combinations of source
All three primary sources 40%
Two sources 50%
One source 10%
Additional sources
Federal loans & Pell Grants (NSLDS) 50%
22
Allowing for Redefining a Case Allowing for Redefining a Case • Student case is a “complete” if valid data exist for:
-- Student type
-- Birth date or age
-- Gender,
-- And at least 8 of the 15 variables:
• Dependent status, marital status, any dependents, income, expected family contribution, degree program, class level, first- time beginner, months enrolled, tuition, received financial aid, received non-federal aid, student budget, race, and parent education.
23
Table 19. Students requiring intensive tracing procedures, by institutional characteristics and student type: 2004
Cases requiring intensive tracing efforts
Institutional characteristics and student type1 Total Number Percent
Total 101,100 25,940 25.7 Institutional level Less-than-2-year 10,350 3,730 36.0 2-year 37,780 11,920 31.5 4-non-doctorate-granting 20,640 4,350 21.1 4-year doctorate-granting 32,320 5,940 18.4 Institutional control Public 65,590 16,950 25.8 Private, not-for-profit 21,670 4,240 19.6 Private, for-profit 13,840 4,750 34.3 Type of institution Public less-than-2-year 2,150 630 29.2 Public 2-year 32,570 10,260 31.5 Public 4-year non-doctorate-granting 8,890 1,910 21.4 Public 4-year doctorate-granting 21,970 4,160 18.9 Private not-for-profit 2-year-or-less 7,570 2,870 37.9 Private not-for-profit 4-year non-doctorate-granting 8,880 1,680 18.9 Private not-for-profit 4-year doctorate-granting 10,060 1,740 17.3 Private for-profit less-than-2-year 7,570 2,870 37.9 Private for-profit 2-year-or-more 6,270 1,880 30.0
Difficult-to-Locate Difficult-to-Locate StudentsStudents
24
Table 17. Student locating, by institutional characteristics and student type: 2004
Located
Institutional characteristics and student type1 Total Number Percent
Total 101,010 80,050 79.2 Institutional level Less-than-2-year 10,330 7,030 68.0 2-year 37,750 28,210 74.7 4-non-doctorate-granting 20,630 17,130 83.0 4-year doctorate-granting 32,310 27,690 85.7 Institutional control Public 65,540 52,360 79.9 Private, not-for-profit 21,660 18,140 83.7 Private, for-profit 13,820 9,550 69.2
Even so, Some Are Even so, Some Are Locatable Locatable
25
Example –B&B:08/09 Self-Administered Web Example –B&B:08/09 Self-Administered Web InterviewInterview
26
Student Mode Choice for InterviewStudent Mode Choice for Interview
Completed interviews
Number Weighted percent
Total 62,220 100.0
Self-administered 28,710 46.7
Early response 17,100 27.5
With prompting 11,610 19.2
Interviewer-administered 33,510 53.3
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).
Almost ½ respond via web. 1/3 receive incentive. Growing trend.
27
NPSAS:04 Experience with IncentivesNPSAS:04 Experience with IncentivesC u m u la tive In te rv iew s C o m p le te d b y W e e k , Im p a c t o f O ffe r in g $ 3 0 In c e n tive , N P S A S :0 4 F u ll-s c a le
C u m ula tiv e C AT I/S e lf-A dm in is te re dIn te rv ie w s
01,00 02,00 03,00 04,00 05,00 06,00 07,00 0
8,00 09,00 0
1 0,00 01 1,00 01 2,00 01 3,00 01 4,00 01 5,00 01 6,00 01 7,00 01 8,00 01 9,00 02 0,00 02 1,00 0
2 2,00 02 3,00 02 4,00 02 5,00 02 6,00 02 7,00 02 8,00 02 9,00 03 0,00 03 1,00 03 2,00 0
Cu
mu
lati
ve
In
terv
iew
s
Cum ula tive C ATI In ts
Cum ula tive Se l f-Adm in In ts
$ 3 0 in c e n tiv e
N o $ 3 0 in c e n tiv e
N o $ 3 0 in c e n tiv e
28
““Incentivized” RespondentsIncentivized” RespondentsStudent Interviews Completed, by Early Completion and Mode: Comparison of Outcomes after 14 Weeks, 2004 and 2008
STUDY YEAR
NPSAS 2004
NPSAS 2008
N
% of target sample
N
% of target sample
Target Sample 120,000 138,000 Total Interviews 26,529 22% 48,921 35% Early Completion Interviews (within 1st 3 weeks)
13,234 11% 35,310 26%
Mode Self-Admin Web Interviews 15,068 13% 38,739 28% CATI Interviews 10,494 9% 9,235 7% No Incentive Interviews 10,142 8% 8,289 6%
$20 vs. $30
Web rises in preferred choice
29
Student Interview Data CollectionStudent Interview Data Collection
Challenge: Minimize bias from case-level and item-level missing data.
Innovations:
• Made weight adjustment at case or student-level
• Imputed item-level missing data by:
-- Logical imputation or
-- Statistical imputation with nearest similar neighbor donor
30
Table 21. Student interview completion results, by institutional characteristics and student type: 2004
Completed interviews1
Institutional characteristics and student type2 Eligible
students3 Number Unweighted
percent Weighted
percent
Total 99,450 62,220 62.6 70.6 Institutional level
Less-than-2-year 10,210 4,830 47.3 50.2 2-year 37,130 20,790 56.0 69.3 4-non doctorate-granting 20,340 13,840 68.0 70.8 4-year doctorate-granting 31,770 22,760 71.6 73.9
Institutional control
Public 64,520 40,620 63.0 71.3 Private, not-for-profit 21,290 14,620 68.7 71.8 Private, for-profit 13,640 6,970 51.1 60.4
Type of institution
Public less-than-2 year 2,130 1,200 56.4 61.6 Public 2-year 31,990 18,000 56.3 69.8 Public 4-non-doctorate-granting 8,760 5,890 67.2 71.9 Public 4-year doctorate-granting 21,640 15,530 71.8 73.8 Private not-for-profit 2-year or less 2,690 1,350 50.3 56.3 Private not-for-profit 4-non-doctorate-granting 8,760 6,250 71.3 70.7 Private not-for profit 4-year doctorate-granting 9,840 7,030 71.4 74.0 Private for-profit less than 2-year 7,470 3,420 45.8 48.6 Private for-profit 2-year or more 6,170 3,550 57.5 65.7
Who Responds and Who Doesn’tWho Responds and Who Doesn’t
31
• Nonresponse bias was estimated using statistical modeling.
• Dependent variable = Responded or Not responded.
• Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.:
-- institution type; region; institution enrollment from IPEDS file (categorical); student type; FTB status; etc.
• Weight adjustments = Coefficients for predictor variables used for weighting.
Nonresponse ProcedureNonresponse Procedure
32
Table 52. Weight adjustment factors for student location nonresponse adjustment: 2004
Model predictor variables
Number of located
respondents Weighted
response rate
Average weight adjustment factor
(WT9)
Total 95,170 95.4 1.07 Type of institution Public less-than-2-year 2,340 95.8 1.70 Public 2-year 29,030 91.7 1.10 Public 4-year non-doctorate-granting 8,490 96.7 1.03 Public 4-year doctorate-granting 20,880 97.0 1.03 Private not-for-profit less-than-4-year 2,680 97.4 1.03 Private not-for-profit 4-year non-doctorate-granting 9,120 98.7 1.03 Private not-for-profit 4-year doctorate-granting 9,310 97.7 1.04 Private for-profit less-than-2-year 7,740 96.1 1.07 Private for-profit 2-year or more 5,590 97.8 1.04 Representative state institution No 56,880 96.8 1.07 Yes 38,290 93.1 1.08 Bureau of Economic Analysis Code (Office of Business
Economics [OBE]) Region1 New England 5,520 96.7 1.05 Mid East 14,630 96.2 1.10 Great Lakes 14,350 96.4 1.05 Plains 7,440 95.1 1.07 Southeast 22,570 96.8 1.05 Southwest 10,410 97.3 1.04 Rocky Mountains 3,760 98.1 1.13 Far West 14,260 89.4 1.13 Outlying Areas, including Alaska and Hawaii 2,230 94.6 1.06
Example Weight AdjustmentsExample Weight Adjustments
Already seen -- hard to locate and under- respond.
So, “double-up” responding case.
33
Imputation ProcedureImputation Procedure• Logical – use other data sources to determine missing
variable values for given case (e.g., substitute FAFSA for institution CADE).
• Statistical –separate cases into dissimilar groups (using a defined set of variables) such that respective group members are alike.
• Membership in final group determines donor candidates or “nearest neighbor.”
• For cases having missing variable values, such values are “borrowed” from the “nearest neighbor.”
34
Table 43. NPSAS:04 interview item nonresponse for items with more than 10 percent missing
Interview section Variable name Description
Number asked
Percent “don’t know”
Percent blank
Total percent
nonresponse1
N4PRBA Earned bachelor’s while a first-professional student
1,510 † 13.0 13.0 Section A: Eligibility and Enrollment N4MAJ2A Major-secondary string 1,420 † 10.8 10.8
N4MAJ2B Major-secondary category 1,420 † 9.8 9.8 N4LT30 Age: less than 30 390 † 17.7 17.7
N4SCH2 School 2 name 540 0.0 13.9 13.9 N4CT2 School 2 city 540 0.0 11.3 11.3 N4LEVL2 School 2 level 540 0.2 15.0 15.2 N4CTRL2 School 2 control 540 0.0 15.4 15.4
N4TASSM Teaching assistantship amount 1,240 † 9.6 9.6 Section B: Financial Aid N4RASSM Research assistantship amount 1,240 † 9.6 9.6
N4TRNSM Traineeship amount 130 † 21.3 21.3 N4GASSM Other graduate assistantship amount 340 † 13.1 13.1 N4STAMT State grant/scholarship amount 8,310 † 13.0 13.0
N4AMNEMP Amount of employer aid 3,960 † 11.2 11.2 N4AMNVET Amount of veteran’s benefits 1,610 † 18.5 18.5 N4AMNPMP Amount of parents’ employer aid 1,080 † 16.6 16.6
N4HOPE Claim Federal Hope scholarship 59,220 31.9 5.4 37.3 Section C: Expenses N4DEDUCT Claim tuition tax deduction 59,250 33.6 3.7 37.3
N4LFLNG Claim lifetime learning tax credit 59,070 33.0 4.9 37.8 N4PARNC Parents income in 2003 40,210 12.6 1.7 14.3
N4TRIBE State/federally recognized tribe 1,380 † 13.1 13.1 Section E: Background N4RACES Race: other specify 6,870 † 17.6 17.6
N4SERCS Service: other specify 440 † 20.3 20.3
N4NEEDS Needs: other specify 510 † 32.9 32.9
† Not applicable.
1 Item nonresponse rates were calculated based on the number of student interview respondents for whom the item was applicable and asked. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).
Item-Level Missing DataItem-Level Missing Data
35
Future Challenges to Data Future Challenges to Data CollectionsCollections
• Increased concerns about security / confidentiality -- both institutions and students
• Reduced access to students
--Increased liability concerns of institutions to release student-level contact information
--Greater cell phone usage
• Raised respondent expectations– offering $ incentives
36
ProductsProducts• Data for analysis
Data analysis system,
http://nces.ed.gov/dasol
Restricted data file, http://nces.ed.gov/pubsearch/licenses.asp
• Reports
On methods … http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006180
On content … http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006186
37
How to be Informed of ProductsHow to be Informed of Productshttp://ies.ed.gov/newsflash
38
Contact InformationContact Information
202-502-7643
202-502-7387
202-502-7383
