Upload
andra-griffith
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
Why Do College Coaching Interventions Work?
Scott Carrell, UC Davis and NBER
Bruce Sacerdote, Dartmouth College and NBER
1
2
US College Completion Ranks 11-13thth In OECD
00.2
0.4
0.6
1970 1980 1990 2000Year
US NZ UK SWIITA FIN SWE
Cascio Gordon Clark JEP 2008
3
US Dept of Ed Trio Programs
$879 million in FY 2011 GEAR up, Upward Bound, Talent Search
Fundamental tenets: Catch students early..by 8th grade to get them college ready Avoid discussion of randomized evaluation Bruce and Scott don’t know what they are talking about
4
US Economist/Ed School Programs
Send letters to top students (Hoxby and Turner) Auto fill the FAFSA (Bettinger Long Oreopolous
Sabonmatsu) Send text messages (FAFSA reminders or deadline
reminders) (Castleman and Page) Offer remote coaching or application help (Phillips and
Reber) Carrell and Sacerdote: in person help, financial
incentives, “likely letters” Clearly different populations in all of these
5
2010 Cohort: Frequency Counts 10th Grade Math Scores for Non College Goers
05
001
000
150
02
000
250
0F
req
uenc
y
1100 1120 1140 1160math_score
05
00
10
00
15
00
20
00
25
00
Frequency
1100 1120 1140 1160 1180math_score
No College College
6
The Mission
Ask whether simple interventions (bonuses, mentoring, likely letters) very late in the game can have a meaningful impact on long term outcomes Attending, persisting, graduating, income Use Clearinghouse data to track into College
Are these marginal college going students very different than the average?
For whom does it work? Try to infer mechanisms using baseline interests, preparation, personality measures SAT Questionnaire, survey data
7
The Answer
Yes If you are female Or a guy who has not taken the SATs If you do not have strong parental and teacher support for
completing applications If you are not extraverted
What doesn’t work Likely letters Letters of encouragement (Delaware, NH) Cash Bonuses alone
8
Hypotheses
Behavioral Fear of process Lack of easily obtained information Procrastination/disorganization
Largely rational: have already optimized
Need expert/ adult help
9
What is the Intervention
Guidance counselors identify high school seniors at risk of not applying to college Ideally students who have expressed interest in applying But have taken few steps Typically identify in mid-December
We randomly choose half for the treatment group Guidance staff then invites treatment group to participate
10
What is the Intervention (2) We visit once per week for 3-5 weeks
– Pair up each HS student with a Dartmouth student– 1.5-4 hours per session – Start and complete applications– Send transcripts as needed– Pay application fees– Complete common apps, two year schools– Start the FAFSA– Sign up for SATs & send scores
Incentives– Free coaching– App fees paid – $100 cash bonus upon completion– Get out of class– Pizza
The Process
11
Mission Accomplished (?)
12
13
Challenges Faced By HS Students And Guidance Counselors
Horrible internet access and computers– In first year, many schools had BLOCKED the application
sites for all NH Community Colleges– When that was fixed, we still had credit card problems– In some cases we brought our own laptops and MiFi cards
Students and parents unfamiliar with Common App, FAFSA, online forms
Students were often selected to be the procrastinators
14
Treatment and Control Standardized Reading Scores
0.1
.2.3
.4.5
kden
sity
re
adin
gST
D
-4 -2 0 2x
Treatment Control
15
Standardized Math Scores: Treatment Group Versus All Non Experimental
0.1
.2.3
.4.5
kden
sity
ma
thS
TD
-4 -2 0 2 4 6x
Treatment All Other
Control Group
Mentoring Treatment
Variable Obs Mean Std. Dev Obs MeanStd. Dev
Accepted Treatment 902 0 0 871 0.454 0.49810th Grade Math Score (Standardized) 798 -0.480 0.937 778 -0.286 0.94310th Grade Reading Score (Standardized) 799 -0.436 0.928 772 -0.278 0.966Math > 50th Percentile in State 798 0.312 0.464 778 0.335 0.472Reading > 50th Percentile in State 799 0.350 0.477 772 0.398 0.490Math >75th Percentile 798 0.164 0.371 778 0.185 0.389Reading > 75th Percentile 799 0.213 0.410 772 0.224 0.417Free and Reduced Lunch Eligible 902 0.277 0.448 871 0.286 0.455Male 902 0.548 0.498 870 0.575 0.495Non-white 902 0.173 0.378 871 0.201 0.401Any College (Clearinghouse) 902 0.438 0.496 871 0.592 0.492Four Year College (Clearinghouse) 902 0.169 0.375 871 0.276 0.447Persist for First Two Years Post Grad 902 0.195 0.397 871 0.240 0.427Persist in a Four Year College 902 0.094 0.292 871 0.115 0.319Enrolled 3+ Semesters
902
0.237
0.426
871
0.292
0.455 Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts.
Table 1: Summary Statistics for Treatment and Control Groups
17
Table 2: Show Treatment Status Unrelated to Pre-Treatment Characteristics
Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. Regressions include birthyear*cohort dummies to control for students' age within grade.
(1) (2)
Treatment Status Men
Treatment Status Women
Standardized 10th Grade Math Score 0.001 0.041
(0.012) (0.025)
Standardized 10th Grade Reading Score -0.025+ -0.006
(0.014) (0.020)
Free Reduced Lunch Eligible -0.043 0.073
(0.027) (0.046)
Student is Nonwhite 0.019 -0.038
(0.032) (0.057)
Observations 1216 866R-squared 0.355 0.321F Pre-Treat Variables 1.281 2.109
p-value 0.294 0.098
(1) (2) (3) (4) Apply to
Any College
(Survey)(OLS)
Apply to Any
College (Survey)
IV Estimate
Women Apply to
Any College
(Survey) OLS
Men Apply to Any
College (Survey)
OLS
Mentoring 0.274** 0.375** 0.294** 0.243**Treatment
(0.050) (0.068) (0.049) (0.078)
Observations 859 859 391 468R-squared 0.234 0.309 0.293 0.231
18
Effect of Treatment Status on Applying
Survey data with 50% response rate. Control mean is .72.Standard errors in parentheses, includes high school* cohort f.e. , age controls, male+ significant at 10%; * significant at 5%; ** significant at 1%
(1) (2) (3) (4) (5) Whole
SampleWomen Men Did Not
Take SAT
Took SAT
Effects on Enrollment Any College
Mentoring Treatment (OLS) 0.060** 0.146** 0.007 0.083** 0.035
(0.018) (0.042) (0.025) (0.026) (0.035)
Transcript Only (OLS) -0.005 0.005 0.000 0.035 -0.049 (0.019) (0.034) (0.021) (0.034) (0.035) Mentoring Treatment (IV) 0.133** 0.299** 0.017 0.160** 0.086
(0.041) (0.087) (0.061) (0.047) (0.085)
First Stage for IV Mentoring Treatment 0.463** 0.500** 0.429** 0.511** 0.444** (0.039) (0.044) (0.042) (0.033) (0.070)Observations 2,623 1,114 1,509 1,453 1,170
19
Baseline Treatment Effects on Enrollment in Any College
Outcome variable is a dummy equal to 1 if the student has any enrollment in college including 2 year or four year colleges. Outcome variables are based on the Nation Student Clearinghouse data. Data include 2009-2014cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. ** p<0.01, * p<0.05, + p<0.1
Effects on Enrollment Four Year College
Whole Sample
Women Men Did Not Take SAT
Took SAT
Mentoring Treatment (OLS) 0.057** 0.107** 0.020 0.103** -0.005 (0.018) (0.031) (0.028) (0.026) (0.033)Transcript Only (OLS) 0.001 0.007 0.003 0.002 -0.038 (0.015) (0.022) (0.028) (0.012) (0.030)Mentoring Treatment (IV) 0.125** 0.222** 0.047 0.202** -0.018
(0.037) (0.062) (0.068) (0.048) (0.083)
First Stage for IV Mentoring Treatment 0.463** 0.500** 0.429** 0.511** 0.444** (0.039) (0.044) (0.042) (0.033) (0.070)Observations 2,623 1,114 1,509 1,453 1,170
20
Baseline Treatment Effects on Enrollment in A Four Year College
(1) (2) (3) (4) (5) Women Women Women Men Women No SAT Data Enrolled in
3+ SemestersEnrolled Any College Both School Years
Post Graduation
Enrolled Four Year
College Both School Years
Post Graduation
Enrolled Four Year
College Both School Years
Post Graduation
Enrolled Second Year Conditional on Enrolled First Year
Mentoring Treatment
0.129*(0.053)
0.105*(0.042)
0.097**(0.030)
0.014(0.041)
-0.040(0.066)
Observations 535 535 535 445 263R-squared 0.172 0.123 0.105 0.220 0.165
21
Treatment Effects on Persistence in College (Women)
Outcome variables are four different ways to measure persistence into the second year of college. Sample is limited to women in the 2009-2012 cohorts.
(1) (2) (3) (4) (5) (6)
SAT Questionnaire Measure
Coefficients on Treatment *SAT/
Survey Measure
Coeff on Treatment
Indicator
Coeff on SAT/
Measure
N Mean SAT/Surv indicator regressed
on Male Dummy
Do Not Need Help With Educ Planning
-0.116*(0.059)
0.126**(0.058)
0.049(0.039)
1302 0.829 0.015***(0.004)
Survey Measure
Parents Help With -0.131* 0.118** 0.133*** 724 0.468 0.014College Applications (0.067) (0.045) (0.041) (0.037)
Teacher Helps With -0.165* 0.112*** 0.089 646 0.172 -0.023
College Applications (0.091) (0.030) (0.062) (0.030) Guidance Counselor -0.009 0.0541 0.037 724 0.312 -0.0982**
Helps with College Application
(0.069) (0.037) (0.057) (0.034)
22
Evidence on MechanismsInteraction of Mentoring Treatment with Sources of Assistance on Applications
Dependent Variable is Enrollment in Any College .
(1) (2) (3) (4) (5) (6)Survey Measure Coeff on
Treat*Survey
Measure
Coeff on Treat
Indicator
Coeff on Survey
Measure
N Mean Survey indicator regressed
on Male Dummy
Log (Hourly wage at Age -0.159* 0.545** -0.052 354 2.931 0.307***30 w. only HS Diploma) (0.078) (0.233) (0.099) (0.052) Log (Tuition+Fees -0.019 0.246 -0.023 506 9.052 0.032Community College) (0.036) (0.310) (0.031) (0.091) Log (Tuition Fees NH -0.028 0.343 0.015 502 10.076 -0.040Public University) (0.030) (0.295) (0.023) (0.085) Need College Degree for -0.010 0.027 0.273*** 663 0.777 -0.089***Stated Career Goal (0.112) (0.094) (0.055) (0.032)
23
Interaction of Mentoring Treatment with Beliefs About Wages/ Tuition
Dependent Variable is Enrollment in Any College .
24
Evidence on MechanismsInteraction of Mentoring Treatment with Sources of Assistance on Applications
Dependent Variable is Enrollment in Any College .
(1) (2) (3) (4) (5) (6)
Survey Measure Coefficients on
Treatment*Survey Measure
Coefficient on
Treatment Indicator
Coefficient on
Survey Measure
N Mean Survey indicator regressed
on Male Dummy
Individual Measures
Likes to meet new people -0.305*** 0.280*** 0.150** 530 0.723 -0.096** (0.086) (0.085) (0.055) (0.039)Enjoy Amusement Rides -0.287** 0.259** 0.097 530 0.696 0.031 (0.136) (0.103) (0.087) (0.040)
Composite Measures Meets Deadlines/ Organized
0.083(0.189)
0.030(0.082)
0.096(0.133)
530 0.343 0.011(0.022)
Adventuresome -0.275 0.239 0.144 530 0.657 0.017 (0.179) (0.143) (0.146) (0.021)Self-Esteem -0.097 0.136 0.143 552 0.672 0.007
(0.128) (0.096) (0.092) (0.028)
25
Interaction of Mentoring Treatment with Personality Traits
Dependent Variable is Enrollment in Any College .
26
Interaction of Mentoring Treatment with Personality Traits
Dependent Variable is Enrollment in Any College .
27
Some Takeaways We find a boots on the ground approach is effective
– Appears to compensate for lack of or non take up parental or teacher help
– No evidence that its more effective for people who struggle with deadlines, planning, organization
– Helps students less far along in process, ie non SAT takers– Much more robust effects for the women..lack of effects for men very
much related to labor market opportunities
Can’t find evidence for effects from…– Likely letters/ letters of encouragement– Cash bonuses along..Though cash bonuses may be very important for
takeup– Texting of CCSNH students has not yielded effects– Nor did letter of encouragement to DE graduated seniors
28
More Takeaways We find evidence in favor of optimization story and
need skilled help story
We don’t find evidence for super naïve behavioral story Or that failure to attend is driven by forgetfulness or lack of
organization But parents/mentors could indeed be solving this
We suspect that letter based interventions or simple text reminders may only work with specific groups or in specific contexts
29
Also Mention If Time Huge drop off in participation when we removed the
cash bonus
Men very likely to say they have a job they prefer to college And no evidence in ACS of OLS return to two years of college
Still have low average persistence in both treat and control
Can’t make the Big Five measures work
30
A Very Few Words on Returns to CollegeOreopoulos and Salvanes
31
Caveats on Earnings Returns to College Large number of papers but sources of ID very
different– We haven’t actually shown that the marginal students in
current college going interventions have earnings gains– Bettinger, Gurantz, Kawano and I having trouble finding
earnings effects from winning the Cal-Grant– The one thing we have learned is just how ridiculously noisy
the outcome is
32
Evidence on MechanismsInteraction of Mentoring Treatment with Sources of Assistance on Applications
Dependent Variable is Enrollment in Any College .
33
Do We Affect Type of College Attended? Treatment 0 1 Nashua Community College 25 24 Manchester Community College 15 22 Nhti - Concord's Community College 14 16 Great Bay Community College 11 8 Plymouth State University 2 8 Rivier College 4 7 University Of New Hampshire 7 6 Southern New Hampshire 0 5 Hesser College - Manchester 4 4 Community College Of Vermont 1 3 Colby Sawyer College 1 2 Hesser College -Nashua 3 2 Northern Essex Community College 0 2 University Of Massachusetts At Lowell 0 2 Arkansas State University - Newport 0 1 Pennsylvania State University 0 1 Total Shown 87 113 Total in College 114 130
34
Do We Affect Type of College Attended? (2)
(1) (2) (3) Institution Attended
is Masters Degree Granting: Both
Genders At Any College
Institution Attended is
Masters Degree Granting: Women
At Any College
Acceptance Rate of College
Attended: Both
Genders Treatment 0.002 0.038 -2.237
(0.040) (0.081) (2.811)
Observations 383 171 145 R-squared 0.133 0.231 0.106
Outcome variables measured in IPEDS data. Sample only includes students in college and for whom we have IPEDS data
35
Split Sample By Test Score
(1) (2) (3) (4) Enrollment
Two Year College Women
Enrollment Four Year
College Women
Enrollment Two Year
College Men
Enrollment Four Year
College Men
Mentoring Treatment 0.137+
(0.067) 0.064
(0.042) 0.014
(0.057) 0.060
(0.041)
Reading Score > 50th Percentile in Treatment Group
-0.208* (0.090)
0.116* (0.052)
-0.054 (0.073)
-0.025 (0.047)
Transcript Only Group -0.044
(0.045) 0.003
(0.032) 0.024
(0.037) -0.026
(0.023)
Observations 788 788 1,080 1,080 R-squared 0.124 0.174 0.075 0.231
(1) (2) (3) (4) VARIABLES Women:
Enrolled in Any College
Men: Enrolled in
Any Year College
Women: Enrolled in Four Year
College
Men: Enrolled in Four Year
College Treatment 0.150 0.027 0.202 0.085
(0.074) (0.018) (0.091) (0.032)
Immigrant/ Refugee -0.039 0.057 -0.211 0.099
(0.384) (0.419) (0.077) (0.226)
Immigrant*Treatment 0.002 0.089 -0.071 -0.260 (0.285) (0.219) (0.066) (0.173) Observations 136 144 136 144 R-squared 0.235 0.182 0.134 0.118
36
Interaction of Treatment with Immigration Status
Data are from Manchester West 2010,2011 Cohorts. Sample is roughly 9% immigrants.
37
Mechanisms Bonus? Incentive?
Same or Cross Gender Mentoring?
What do participants say?
More effective in schools with fewer resources?
38
Evidence From 2012 Cohort (Coaching Plus $100 Bonus Versus Bonus Alone)
39
Post-Survey Evidence on $100 Bonus
$100 bonus affect Decision to Complete Program?
Freq.
Aware, no effect
11 Initially motivating not long run factor
2
Not expecting
4 Important
2
Total 19
Take Up Rates Within Mentoring Group
41
What Aspects Helped the Most? 19 of 19 mentioned in person help with applications
5 of 19 mentioned the $100 bonus– 3 of 7 men versus 2 of 10 women
12 of 19 mentioned our paying for application fees– Rational bc a lot more than $100 for some students
Treatment Effect by High School
43
Average College Going Versus Effect Size For Women
44
Average College Going Versus Guidance Counselors Per Student
Manchester WestManchester WestDover
Portsmouth
Pinkerton
Manchester WestManchester WestManchester West
Nashua NorthNashua North
KearsargeKearsarge
LebanonLebanon
Pinkerton
Manchester WestLebanon
Kearsarge
Nashua South
Nashua North
KearsargeKearsarge
Nashua North
Nashua South
Manchester WestManchester West
Nashua NorthNashua North
Pinkerton
Portsmouth
Lebanon
Nashua North
Manchester West
Nashua North
Manchester West
Nashua North
Manchester WestManchester WestManchester West
Nashua South
Manchester WestDover
Manchester WestManchester West
Kearsarge
Nashua South
Portsmouth
Nashua North
Dover
Lebanon
Portsmouth
Nashua North
Dover
Nashua South
LebanonManchester West
Nashua South
Nashua North
Dover
Nashua North
Kearsarge
Nashua NorthNashua North
PinkertonPinkerton
Londonderry
Nashua North
Nashua South
Manchester WestLondonderry
Nashua NorthNashua North
Manchester West
Nashua North
Portsmouth
Manchester West
Nashua North
Manchester WestManchester West
Portsmouth
Lebanon
Kearsarge
Manchester West
Kearsarge
Nashua North
Manchester WestLebanon
Nashua North
Lebanon
Nashua North
Manchester West
Pinkerton
Manchester WestManchester WestManchester West
Nashua NorthNashua North
Pinkerton
Nashua NorthNashua North
Manchester WestManchester West
Nashua NorthNashua North
Manchester WestManchester WestManchester West
Nashua NorthNashua North
Lebanon
Nashua North
Nashua South
Manchester WestManchester West
Kearsarge
Dover
Portsmouth
Manchester West
Nashua NorthNashua North
DoverDoverManchester West
Kearsarge
Portsmouth
Manchester WestManchester West
Pinkerton
Nashua North
Manchester West
Nashua North
Manchester WestManchester West
Pinkerton
Nashua North
Portsmouth
Manchester West
Kearsarge
Portsmouth
Kearsarge
Manchester West
Nashua North
Manchester WestManchester WestLebanon
Pinkerton
Portsmouth
Manchester WestLondonderry
Pinkerton
Manchester West
Pinkerton
Nashua South
Nashua North
DoverDoverPinkerton
Nashua South
Nashua North
Pinkerton
Nashua South
LebanonManchester West
Nashua North
Portsmouth
Nashua NorthNashua NorthNashua North
Manchester WestDover
Nashua North
Dover
Nashua North
Kearsarge
Manchester West
Nashua SouthNashua South
Manchester West
Nashua South
Dover
Nashua NorthNashua NorthNashua North
Nashua SouthNashua South
Nashua North
Nashua South
Nashua North
Nashua South
DoverManchester West
DoverManchester WestManchester WestManchester West
Nashua South
Dover
Lebanon
Nashua SouthNashua South
Manchester West
Pinkerton
Manchester WestLebanon
Manchester WestManchester West
Nashua North
Manchester West
Nashua North
Manchester West
Nashua South
Manchester West
PinkertonPinkerton
Nashua South
PinkertonDover
Nashua North
Nashua South
Portsmouth
Nashua SouthNashua South
Lebanon
Nashua South
Nashua NorthNashua North
Pinkerton
Nashua South
Nashua NorthNashua North
Londonderry
Nashua NorthNashua NorthNashua North
Nashua South
Dover
Nashua NorthNashua North
Nashua SouthNashua South
DoverPinkerton
Portsmouth
Manchester West
Nashua SouthNashua South
Pinkerton
Nashua NorthNashua NorthNashua North
Nashua SouthNashua South
Nashua North
Manchester WestLebanon
Nashua South
Portsmouth
Manchester West
Nashua South
Manchester WestManchester West
Portsmouth
Nashua South
Nashua North
Nashua South
Nashua NorthNashua NorthNashua North
Nashua SouthNashua SouthNashua SouthNashua South
Manchester West
Nashua North
Kearsarge
Nashua NorthNashua NorthNashua North
Pinkerton
Manchester WestManchester West
Nashua South
Nashua North
Nashua South
Manchester West
Nashua North
Nashua South
Manchester West
Pinkerton
Manchester West
Pinkerton
Nashua North
Manchester West
PinkertonPinkerton
Lebanon
Nashua North
Lebanon
Nashua SouthNashua South
Nashua North
Manchester West
PinkertonDover
Nashua South
Lebanon
Pinkerton
Nashua SouthNashua South
Kearsarge
Nashua SouthNashua South
Pinkerton
Manchester West
Nashua North
Nashua South
Nashua North
Nashua South
Pinkerton
Manchester West
Pinkerton
Portsmouth
Kearsarge
Nashua South
Manchester West
Nashua SouthNashua South
Nashua North
Manchester West
Nashua SouthNashua South
Nashua NorthNashua NorthNashua NorthNashua North
Manchester West
Nashua South
Nashua North
Lebanon
Pinkerton
Londonderry
Kearsarge
Manchester West
Kearsarge
Nashua South
Kearsarge
Nashua SouthNashua South
Manchester WestManchester West
PinkertonPinkertonPinkerton
Nashua South
Kearsarge
Manchester West
Nashua South
Manchester WestManchester West
Nashua South
Portsmouth
Nashua South
Nashua NorthNashua North
Manchester West
Pinkerton
Nashua NorthNashua North
Nashua South
Lebanon
Kearsarge
Manchester West
Nashua North
Lebanon
Pinkerton
Nashua South
Manchester West
Nashua South
Kearsarge
Manchester West
Nashua North
Kearsarge
Nashua SouthNashua South
Nashua North
Nashua SouthNashua South
DoverPinkerton
Nashua NorthNashua North
Pinkerton
LondonderryManchester West
Nashua North
PinkertonPinkerton
Nashua South
Pinkerton
Nashua South
Nashua NorthNashua NorthNashua NorthNashua North
Pinkerton
Nashua South
Pinkerton
Nashua NorthNashua NorthNashua NorthNashua NorthNashua North
Nashua South
Nashua North
Manchester West
Nashua South
Nashua North
Manchester WestManchester West
Kearsarge
Pinkerton
Portsmouth
DoverManchester West
Nashua South
Manchester West
Pinkerton
Nashua South
Manchester West
PinkertonPinkerton
Nashua South
Manchester WestManchester WestManchester WestManchester WestManchester West
Nashua North
Manchester West
Nashua South
Nashua North
Pinkerton
Manchester West
Nashua South
Manchester West
Nashua South
Nashua North
Manchester WestManchester West
Nashua North
Manchester West
Kearsarge
Pinkerton
Nashua South
Manchester West
Pinkerton
Nashua NorthNashua NorthNashua North
Nashua South
Pinkerton
Nashua SouthNashua South
Portsmouth
Manchester West
Nashua South
Manchester West
Nashua North
Dover
Nashua NorthNashua North
Pinkerton
Nashua SouthNashua South
Portsmouth
Manchester West
Nashua South
DoverPinkerton
Manchester West
Nashua South
Manchester WestManchester West
Pinkerton
Nashua South
Dover
Nashua North
Londonderry
Nashua South
Nashua North
Kearsarge
Manchester WestManchester WestManchester WestManchester West
Pinkerton
Nashua South
Nashua North
Manchester West
Pinkerton
Manchester WestManchester West
Pinkerton
Manchester West
Nashua South
Manchester West
Nashua South
Portsmouth
Manchester West
PortsmouthPortsmouth
LondonderryLebanon
Dover
Nashua North
DoverPinkerton
LebanonManchester West
Dover
LebanonLebanon
Nashua SouthNashua South
Pinkerton
Nashua North
Pinkerton
Nashua SouthNashua South
PinkertonDover
Kearsarge
Nashua South
Manchester WestLebanon
DoverPinkertonPinkerton
Nashua South
Pinkerton
Lebanon
Pinkerton
Nashua NorthNashua North
Manchester WestManchester West
Nashua SouthNashua South
Portsmouth
KearsargeKearsarge
Nashua South
Manchester West
Nashua South
Pinkerton
Kearsarge
Nashua South
PortsmouthPortsmouth
Manchester WestManchester West
PinkertonPinkertonDover
Nashua South
LondonderryManchester WestManchester West
Nashua North
Portsmouth
PinkertonPinkertonPinkerton
Lebanon
DoverDoverDover
Nashua South
Londonderry
Nashua South
Pinkerton
Manchester West
Nashua SouthNashua South
Pinkerton
Nashua SouthNashua SouthNashua South
Manchester WestDover
Manchester WestManchester WestManchester WestManchester WestManchester WestManchester West
Pinkerton
Portsmouth
Manchester WestManchester West
Nashua South
Pinkerton
Manchester WestDover
Manchester West
PortsmouthPortsmouth
Lebanon
Nashua South
PinkertonDoverDover
Manchester West
Nashua South
Pinkerton
Nashua North
Kearsarge
Manchester WestManchester West
Nashua South
Manchester WestManchester West
Nashua SouthNashua South
PinkertonPinkertonPinkertonDover
Nashua South
Manchester West
Pinkerton
Kearsarge
Dover
Nashua South
Dover
Nashua South
Portsmouth
Manchester WestManchester West
Nashua SouthNashua South
Manchester WestManchester West
Nashua South
Portsmouth
Nashua SouthNashua South
Portsmouth
Nashua North
Portsmouth
Pinkerton
Nashua South
Manchester West
Nashua North
Portsmouth
Dover
Nashua South
Manchester West
Nashua South
Pinkerton
Nashua South
Nashua NorthNashua North
Nashua SouthNashua South
Manchester West
Nashua South
Manchester WestManchester West
Pinkerton
Nashua South
Dover
Nashua SouthNashua SouthNashua SouthNashua SouthNashua South
Nashua North
DoverPinkerton
Manchester West
Nashua North
Portsmouth
LondonderryDover
Manchester WestManchester WestManchester WestManchester WestManchester West
Nashua South
Nashua North
Nashua SouthNashua South
Lebanon
Nashua SouthNashua South
Lebanon
Pinkerton
Nashua South
Manchester WestManchester West
Nashua North
Nashua South
Portsmouth
Nashua South
Manchester WestDover
Manchester West
Nashua SouthNashua South
Londonderry
Nashua South
Pinkerton
Nashua NorthNashua North
Portsmouth
Nashua South
Portsmouth
PinkertonPinkertonPinkertonPinkerton
Nashua NorthNashua North
Manchester WestLebanonLebanon
Nashua South
Manchester West
Nashua SouthNashua South
Manchester WestManchester West
Nashua South
PinkertonPinkerton
PortsmouthPortsmouth
Manchester West
Kearsarge
Nashua South
Manchester WestLebanon
Manchester West
Nashua South
Manchester West
Portsmouth
Nashua South
Nashua North
Manchester West
Nashua SouthNashua SouthNashua South
Manchester West
Nashua South
Nashua NorthNashua North
Nashua SouthNashua South
LebanonLondonderry
Dover
Lebanon
Nashua SouthNashua South
Pinkerton
Nashua South
Manchester WestLebanon
Pinkerton
Nashua North
Manchester WestManchester WestManchester West
Nashua South
Portsmouth
Manchester WestManchester WestManchester WestManchester West
Nashua North
Manchester WestLebanon
Nashua North
Manchester WestManchester West
Pinkerton
Manchester West
Nashua South
Manchester WestDover
Nashua South
Pinkerton
Nashua SouthNashua South
Portsmouth
Nashua North
Manchester West
Pinkerton
Nashua South
Pinkerton
Nashua South
Portsmouth
Manchester WestManchester WestManchester WestLebanon
Nashua North
Dover
Nashua North
PinkertonPinkerton
Nashua SouthNashua South
Portsmouth
Nashua NorthNashua North
Nashua SouthNashua South
DoverPinkerton
Nashua North
Manchester West
PinkertonPinkertonPinkerton
LebanonManchester WestManchester West
Nashua South
Manchester WestManchester WestDover
Pinkerton
Londonderry
Pinkerton
Portsmouth
Manchester WestManchester West
Nashua SouthNashua SouthNashua South
Nashua NorthNashua North
Manchester WestManchester West
Nashua North
Manchester West
Nashua North
Nashua South
Nashua North
Nashua South
Manchester West
Nashua South
Manchester WestManchester WestManchester West
Nashua South
Nashua North
Nashua South
Manchester West
Nashua South
Pinkerton
Nashua South
Portsmouth
Manchester WestLondonderryManchester WestDover
Nashua South
Pinkerton
Manchester WestManchester West
Nashua South
Manchester WestManchester West
Nashua North
Manchester WestManchester West
Nashua SouthNashua South
Dover
Nashua SouthNashua SouthNashua SouthNashua SouthNashua South
Nashua North
Manchester West
Pinkerton
Nashua SouthNashua SouthNashua South
Nashua North
Manchester West
Nashua South
PortsmouthPortsmouth
Nashua North
Nashua South
Manchester WestManchester West
Nashua North
Nashua SouthNashua South
Manchester West
Nashua South
Kearsarge
Manchester West
Nashua South
Manchester West
Pinkerton
Nashua South
Nashua North
Nashua SouthNashua SouthNashua South
Nashua NorthNashua North
Nashua South
Manchester West
Nashua South
Nashua NorthNashua North
Manchester West
Nashua South
Nashua NorthNashua North
Pinkerton
Nashua South
Nashua North
Nashua South
PinkertonDover
LebanonManchester West
Portsmouth
Nashua SouthNashua South
Manchester West
Nashua South
Manchester WestManchester West
Nashua SouthNashua SouthNashua South
DoverManchester West
Nashua SouthNashua South
Manchester West
Nashua North
Pinkerton
Nashua SouthNashua SouthNashua SouthNashua South
Manchester West
Nashua SouthNashua South
Nashua North
Nashua South
Nashua North
Pinkerton
Nashua North
Nashua South
DoverManchester West
Nashua South
Portsmouth
PinkertonPinkerton
LebanonManchester West
Pinkerton
Manchester WestManchester West
Nashua South
DoverPinkerton
Nashua SouthNashua SouthNashua South
Pinkerton
Nashua South
Londonderry
Nashua South
Nashua North
Pinkerton
Nashua SouthNashua SouthNashua SouthNashua SouthNashua South
Pinkerton
Manchester West
Nashua South
Nashua North
Pinkerton
Manchester West
Nashua SouthNashua SouthNashua South
Pinkerton
Nashua South
Dover
Lebanon
Pinkerton
Manchester West
Pinkerton
Nashua SouthNashua South
Nashua North
Pinkerton
Manchester WestLebanon
Manchester West
Pinkerton
Portsmouth
Pinkerton
Manchester WestManchester WestLebanon
Nashua South
Manchester WestManchester WestManchester West
PinkertonPinkerton
Portsmouth
Manchester WestManchester WestManchester WestManchester West
PinkertonDover
Nashua North
Nashua South
Nashua North
Manchester West
Nashua SouthNashua South
Pinkerton
Manchester West
Nashua North
Nashua SouthNashua South
Manchester West
Nashua South
Manchester WestManchester WestManchester West
Pinkerton
Nashua SouthNashua South
Pinkerton
Manchester WestManchester WestManchester West
Nashua SouthNashua SouthNashua South
Pinkerton
Manchester West
PinkertonPinkerton
Nashua South
Nashua North
Kearsarge
Manchester West
Nashua North
Nashua South
Manchester WestDover
Nashua SouthNashua South
Manchester West
Nashua South
Dover
Nashua North
Manchester West
Pinkerton
Nashua South
Portsmouth
Nashua South
Kearsarge
Manchester WestManchester West
Nashua South
Nashua North
Nashua South
Portsmouth
Nashua North
DoverDoverManchester WestManchester WestManchester West
Nashua SouthNashua South
Pinkerton
Manchester WestDover
Pinkerton
Nashua SouthNashua South
Pinkerton
Kearsarge
Lebanon
Pinkerton
Nashua South
DoverDover
Nashua South
PinkertonDover
Nashua South
Pinkerton
Nashua South
Dover
Nashua South
Manchester WestManchester WestManchester WestManchester West
Nashua North
Nashua SouthNashua South
PinkertonDoverDover
Pinkerton
0.1
.2.3
.4e
stim
ate
d_e
ffect
.003 .004 .005 .006guidance_per_student
Mentoring Treatment Interaction with Sources of Disadvantage
Mentoring Treatment Interaction with Sources of Disadvantage
Mentoring Treatment Interaction with Sources of Disadvantage
48
Stories from the HS Students > hey Cambell, its Daniel M from west, one of the
student you hepled apply for college. If i remember correctly, you told me to let you know what college i get into. Well, i was recently accepted to NHTI in Concord.
There are still some things i need to take care of, but its nothing serious.
> P.S. Thanks for you help. Sorry for all the flirting
49
Cross Tab of Student Male and Assigned Male Mentor
assigned_male_mentor
Student is
Total Male 0 1
0 76 73 149 1 61 95 156
Total 137 168 305
Mentors were assigned on a first come first served basis, but when multiple arrivals occurred at the same time, we had a modest bias towards same gender pairings. Regressions include a dummy for being assigned to treatment but not showing up to be assigned a mentor.
Is Cross Gender Mentoring More Effective?
51
Four Theories As To Why It Only Works for Women
Treatment is actually a complement to ability or persistence– Look for other ways in which treatment might interact with sources
of advantage, help or test scores
Guys are primed to either be near the top of an endeavor or quit– Don’t respond well to feedback that they are in the middle
Labor economics – Short run returns to college in NH are lower for guys
Are guys negatively selected into the treatment group? – In part bc of $100 bonus?
52
Survey Data on Male And Female Study Subjects
Women Men N Mean SD N Mean SD No Sat Data
1077 0.57 0.49 1463 0.57 0.50
Sat Math
95 42.77 8.33 133 45.42 9.38
Sat Verbal
95 44.02 8.48 133 45.14 10.47
Role Model College
43 0.16 0.37 61 0.21 0.41
Self Worth High
45 0.51 0.51 67 0.60 0.49
Remember Dates
44 0.30 0.46 67 0.31 0.47
Forget Deadlines
44 0.32 0.47 67 0.30 0.46
Have Control In Life 45 0.56 0.50 67 0.54 0.50
53
Survey Data on Male And Female Study Subjects (Aspirations and Expectations)
Women Men N Mean SD N Mean SD Aspire College4 46 0.80 0.40
68 0.81 0.40
Role Model College 43 0.16 0.37
61 0.21 0.41
Sat Aspire College4 95 0.65 0.48
133 0.65 0.48
Teachers Expect College 43 0.81 0.39
66 0.71 0.46
Parents Want College4 43 0.67 0.47
67 0.52 0.50
Talked Future Plans Teacher 46 0.46 0.50
68 0.43 0.50
Talked Future Plans Parent 46 0.85 0.36
68 0.85 0.36
(1) (2) (3) (4) Log Total
Income Men NH
Log Total Income
Women NH
Log Total Income Men
All Other States
Log Total Income
Women All Other States
High School 0.399** 0.409** 0.429** 0.477**
(0.055) (0.103) (0.003) (0.005)
One to Three Years of 0.686** 0.667** 0.726** 0.761** College
(0.058) (0.104) (0.003) (0.005)
Four Plus Years of 1.081** 0.951** 1.251** 1.252** College
(0.055) (0.102) (0.003) (0.005)
Constant 9.955** 9.325** 9.748** 9.146** (0.051) (0.099) (0.003) (0.004) Observations 4,674 4,570 1,129,025 1,050,794 R-squared 0.140 0.049 0.168 0.106 F Test HS=Some College
77.69 33.52 15889 9056
p-value 0 7.53e-09 0 0
54
How Do Returns to College Differ for Men Versus Women in NH (Ages31-40)?
55
How Do Returns to College Differ for 21-30 year old Men Versus Women in NH?
(1) (2) (3) (4) Log Total
Income Men NH
Log Total Income
Women NH
Log Total Income
Men All Other States
Log Total Income
Women All Other States
High School 0.343** 0.403** 0.345** 0.484**
(0.075) (0.100) (0.004) (0.005)
One to Three Years 0.339** 0.593** 0.405** 0.673** of College
(0.078) (0.101) (0.004) (0.005)
Four Plus Years of 0.663** 0.848** 0.839** 1.193** College
(0.077) (0.099) (0.004) (0.005)
Observations 2925 2898 828,881 794,172 R-squared 0.033 0.046 0.055 0.095 F Test HS=Some College
0.00493 14.49 414.6 3331
p-value 0.944 0.000144 0 0
56
Other Interventions Find Effects Differ By Gender?
Moving to Opportunity only saw decreases in crime and increases in school engagement for girls
Angrist and Lavy bonuses in Israel for obtaining Bagrut
Study one of Oreopoulos and Angrist bonuses plus mentoring for U of Toronto students– Study two did not find any effects
Dynarksi Ichino find bigger effects of offereing finacial aid for women
Job Training programs affect women more
57
Other Interventions Find Effects Differ By Gender?
But not at all clear this is true for charter school studies– Boston area charter studies find bigger effects for boys
Are effects bigger for students who are weaker in that domain– Did boys already attend college up to zero marginal returns?– Girls need a shot of confidence or more help
Or is the treatment a complement to skill?– Is it ultimately about personality measures, follow through– Our intent is to collect ex-post survey measures of personality
traits, patients, follow through
58
Additional Interventions Send one group a letter offering admission to their
closest public college and also community college– We’ve had fantastic cooperation from admissions officers at
NH institutions (UNH, UNH Manchester, SNHU)– Not much evidence of an effect yet
Compare to Hoxby Turner which has large effects from letters sent to students in top 10% of SAT / ACT distribution– Our population is very different and perhaps needs much
more in person help hand-holding
59
Additional Interventions (2) Have students meet with Admissions Officers from
Community College System
Have Lets Get Ready run the program. Adds a serious SAT component Ask how much results change when program is scaled up/ PI is
no longer involved And when population of students is broadened
60
Additional Interventions (3) Use text messaging to Community College students to
prevent Summer Melt, encourage persistence, help students make the transition to 4 year institutions
61
Bottom Lines Big differences by gender..critical to better understand that..Quite
possibly its not a general boy phenomenom
These marginal students persist at same rate as average
Program is enormously successful for immigrants– And for resource poor high schools– Suggests that we are in some way compensating for missing component
rather than reinforcing advantage
Difficult or impossible to evaluate this sort of program using endogenous take up
May be real downsides to offering cash bonuses May be real upsides to “expensive” in person help
62
Could We Estimate the Effects for Women Without a Randomized Control Group?
(1) (2) (3) (4) Women
Four Year College:
Intended Treatment Versus All Nonexperi
mental
Women Any
College: Intended
Treatment Versus All Nonexperi
mental
Women Four Year
College: Intended
Treatment Versus All Nonexperi
mental
Women Any
College: Intended
Treatment Versus All Nonexperi
mental Assigned to Treatment Group -0.162** -0.004 (0.030) (0.034) Treated -0.138** -0.041
(0.040) (0.040)
Observations 21,042 21,044 20,934 20,936 R-squared 0.273 0.170 0.272 0.171
63
Compare to Existing Work HR Block (FAFSA) study
– The Role of Simplification and Information in College Decisions: Results from the H&R Block FAFSA Experiment• Eric P. Bettinger, Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu
– Moved enrollment rates from 26.8 to 34.5 percent– 7 percentage point effect in same age group– We have a more intensive intervention and (arguably/ debatably) twice the effect
size
COACH program (Avery and Kane)
Kane college mentoring in CA– 5 percentage point effect on enrollment rates
Berman Ortiz and Bos (ongoing in LA Unified) Phillips and Reber remote coaching in LA and CA
64
Estimate the Effects By Comparing Those That Accept Treatment to Those That Don't?
(1) (2) (3) (4) Women 4
Year College: Accepted
Treatment Versus Didn't
Women Any College:
Accepted Treatment
Versus Didn't
Men Four Year College:
Accepted Treatment
Versus Didn't
Men Four Year College:
Accepted Treatment
Versus Didn't Accepted Treatment 0.076 -0.035 0.070+ 0.017
(0.091) (0.092) (0.035) (0.067)
Constant 0.613* 0.228 -0.477** -0.601** (0.244) (0.466) (0.138) (0.040) Observations 259 259 341 341 R-squared 0.167 0.213 0.176 0.231
65
Modifications in Follow Up Work Try W/o the $100 bonus
– May actually increase treatment on treated for men– Eliminate noise of people signing up for wrong reason
Have Lets Get Ready run the intervention– Include SAT prep. Include juniors
Try letters of encouragement from college admissions departments
Extensive pre and post surveys to learn more about students personalities, career plans, stick-toitiveness
Try emails and texts to spur students to apply
66
Partnered with NH DOE NH DOE has been very supportive and flexible
– Commissioners and staff have leant their support to the project
Key pre-treatment and outcome variables collected by NH DOE– Nat’l Student Clearinghouse subscription– Test scores and demographics
Part of a larger Data Warehouse project underway
Future projects being designed jointly
What our High Schools Look Like
67