Nudging to No Effect: Attempts to Improve the College ... 2017 AEFP paper.pdf · Nudging to No Effect: Attempts to Improve the College Entrance Exam Attendance Rates of Low-Income

Running Head: NUDGING TO NO EFFECT 1

Nudging to No Effect: Attempts to Improve the

College Entrance Exam Attendance Rates of Low-Income Students

Ty M. Cruce

ACT, Inc.

Robert W. Hahn

University of Oxford

Robert D. Metcalfe

University of Chicago

Author Note:

This manuscript is a working paper. Please do not cite without author’s permission. Please

address all correspondence about this study to Ty Cruce ([email protected]).

NUDGING TO NO EFFECT 2

Abstract

Taking a college entrance exam such as the ACT or SAT can be an important step in the college-

going process. Despite the benefits of taking a college entrance exam, a large and

disproportionate share of students who register for the exam with a fee waiver do not

subsequently attend on their registered test date. To address this discrepancy, ACT created it’s

“Dialing for Scholars” research initiative with the goal of increasing the test-taking rate for

students who registered with fee waivers by reaching out to them in the weeks before their

scheduled national ACT test dates to proactively share information that would better prepare

them for the testing experience and encourage them to attend. We designed nine variations on an

intervention that would remind fee waiver students of their test date and “nudge” them to attend.

Despite having ample statistical power, we found that none of the nine interventions tested

across the two studies had a statistically significant impact on the attendance rates of treated fee

waiver students when compared to fee waiver students in their respective control groups. There

are several possible reasons for why our treatments were not effective, and those reasons point to

future opportunities for research.

Keywords: Randomized controlled trial, test taking, low income students


Nudging to No Effect: Attempts to Improve the

College Entrance Exam Attendance Rates of Low-Income Students

Taking a college entrance exam such as the ACT or SAT can be an important step in the

college-going process. According to the most recent data from the National Center for Education

Statistics (2015), 1,617 postsecondary institutions across the US that educate approximately

eight million degree-seeking undergraduates either require or recommend that test scores be

provided as a criterion for admissions decisions. Moreover, 76% of degree-granting four-year

public colleges and 68% of degree-granting four-year private not-for-profit colleges either

require or recommend test scores for admissions. Sitting for the ACT or SAT satisfies this

criterion and thus increases the postsecondary educational options that are available to students

directly after high school.

Taking a college entrance exam can also expand the number and type of postsecondary

institutions that students consider during the search stage of the college choice process by

providing an opportunity for students to be identified and recruited by institutions that are not

currently under consideration. At the time that students register for the ACT or SAT, they may

opt into a service such as ACT Inc.’s Educational Opportunity Service (EOS) or The College

Board’s Student Search Service. These services allow accredited postsecondary institutions to

identify and select the names and contact information (i.e., email address or street address) of

prospective students whose backgrounds and academic achievement levels align with the mix of

desired characteristics of the students in their applicant pools. A recent study (Moore & Cruce,

2017) finds that students who unintentionally opted into EOS had larger college consideration

sets than a matched set of students who intentionally opted out of EOS, suggesting that these


types of search services are having the intended impact of increasing student opportunity by

expanding their awareness of other college alternatives.

Finally, regardless of whether a student wants to begin their college career at a four-year

or two-year institution, taking a college entrance exam can provide the student with important

feedback about their academic progress and readiness for college coursework. Both the ACT and

SAT provide test scores and subscores that allow students to assess their relative strengths and

weaknesses and to identify areas where they could improve. Both organizations also provide

benchmarks by which students can gauge their academic readiness for coursework typically

offered during the first year of college (Allen, 2013; Allen & Sconing, 2005; College Board,

2016; Wyatt, Kobrin, Wiley, Camara, & Proestler, 2011).

Although both ACT, Inc. and The College Board have contracts with a limited number of

states to offer the ACT or SAT statewide, many college-bound students continue to take either of

these exams on one of six to seven national test dates that are offered on Saturdays during the

school year. The current fee to sit for the ACT or SAT is $42.50 ($58.50 with Writing) and $45

($57 with Essay), respectively. These fees may create a financial burden for lower-income

students and their families, thus potentially limiting their access to many postsecondary

institutions across the US. To offset this financial burden and to increase test-taking and

subsequent college attendance among lower-income students, both ACT, Inc. and The College

Board have created programs that provide income-eligible students with fee waivers in order to

register to take their respective test on a national test date. These fee waiver programs are large

in scope and well-utilized, accounting for a sizable share of the total test registrations per school

year. For example, twenty-nine percent of all ACT test registrations for the 2014-15 school year


were completed using a fee waiver. This amounts to $29.2 million offered in fee waivers,

covering over 600,000 ACT registrations.

Within both ACT, Inc. and The College Board’s fee waiver programs, eleventh or twelfth

grade students who are either enrolled in or eligible to participate in the National School Lunch

Program may obtain up to two fee waivers to use for national test day registration.1 The students’

eligibility for a fee waiver is verified by a high school counselor or other designated school

official and the fee waiver is distributed locally. Each fee waiver then applies to the registration

for an individual test date. If the student does not attend on their registered test date, the fee

waiver is forfeited and cannot be applied to a future test date.

Despite the importance of taking a college entrance exam as a step toward increasing

postsecondary educational opportunity, historically, a large and disproportionate share of

students who register with a fee waiver do not subsequently attend on their registered test date.

For example, although overall, 1 out of 10 students who registered for the ACT on a national test

date during the 2014-15 school year were absent on their registered test date, the absentee rate

for students who registered with a fee waiver (at 22%) is substantially higher than the absentee

rate for students who paid the registration fee (at 6%). Even though fee waiver students

comprised 29% of all ACT registrations that year, they represented 61% of all absentees on

national test days. This absenteeism on the part of fee waiver students amounted to

approximately 132,000 empty seats on test days and roughly $6.3 million in forfeited fee waivers

for the academic year.

To address this discrepancy in test day attendance rates, ACT announced a new research

initiative, known as “Dialing for Scholars,” at the White House Summit on College Opportunity

in December, 2014 (ACT, 2014). This initiative was designed with the goal of increasing the test-


taking rate for students who registered with fee waivers by reaching out to these students via

telephone or email in the weeks before their scheduled national ACT test dates to proactively

share information that would better prepare them for the testing experience and to encourage

them to attend.2

The purpose of this paper is to share the methodology and results of two large-scale

experiments that were designed to test the effectiveness of nine different low-cost and low-touch

interventions intended to “nudge” fee waiver students to attend on their test day. The two

experiments described in this manuscript were preceded by a small pilot experiment (Cruce,

2016) that was intended to establish a baseline for future efforts under the Dialing for Scholars

initiative. The question of interest addressed in the pilot experiment was: Does a brief reminder

of the test date and of the required test materials a few days prior to the test date result in fee

waiver students having higher test day attendance rates? The treatment for this study entailed

sending a pre-recorded informational telephone message to a random sample of students a few

days prior to the national test date in which they had registered with a fee waiver. This pilot

design served as a baseline, as it required the least amount of deviation from current ACT

business practices regarding the content, mode, and timing of contacts with students for other test

registration and reporting processes. Using a randomized controlled trial with 2500 students, the

findings of the pilot study did not provide evidence that the pre-recorded telephone reminder, as

delivered, had any impact on the attendance rates of fee waiver students.

There are several possible reasons for why the treatment delivered in the pilot study was

not effective, and these potential reasons point to some of the directions that we took with the

current studies discussed in this manuscript. First, regarding the delivery of the treatment, the

pre-recorded telephone call used in the pilot may not have been the best mode for reaching


students. Specifically, Cruce (2016) did not know who (e.g., parent, student, sibling, or other

relation) received the message and if that message was properly conveyed to the student. In both

of the current studies we send email messages to the students’ email addresses to see if this

change in the mode of delivery would improve the test day attendance rate of fee waiver

students.

Second, a reminder alone in the pilot experiment may not have been sufficient to prompt

students to attend on the test date. It may be the case that some students needed some motivation

to attend in addition (or as opposed) to a reminder. In Study 1 we examine the impact of adding

new content to the reminder that frames their choice by stating either a potential benefit

associated with their attendance on test day or a potential cost associated with their absence.

Finally, in the pilot study Cruce (2016) delivered the reminder only once, a few days

before the test date, which may not have provided students and their families with sufficient time

to overcome potential constraints and prior obligations in order to attend. Low income students

are more likely than their peers to face constraints such as a lack of transportation or family or

work obligations that could keep them from attending (Balfanz & Byrnes, 2012; Criden, 2008).

In Study 2 we add both a timing component and a frequency component to the reminder to

examine whether earlier and more frequent reminders and tips about addressing potential

constraints leads to improved test-day attendance rates.

In the remainder of the manuscript we will provide a review of some of the previous

empirical literature that supported the design of our interventions. We will then give an overview

of the methodology and results of the two experiments conducted during the 2015-16 school

year, and finally we will discuss future directions this research will take.


Literature Review

The interventions designed for the two studies discussed in this manuscript are intended

to provide choice environments that are conducive to helping students make decisions that

improve their well-being while still allowing them the freedom of choice. Thaler and Sunstein

(2008) refer to these types of interventions as “nudges.” Although there is growing list of nudges

and empirical evidence to support their effectiveness in specific choice environments—for a

database of empirical nudges, see http://economicspsychologypolicy.blogspot.com/—in this

study we will address two nudges: reminders and goal framing.

Reminders

Reminders or notices are among the list of nudges currently being used by behavioral

economists and other researchers to help individuals keep their commitments (Samson, 2015;

Sunstein, 2014). Perhaps the area reminders have been most employed is in the health sciences.

Numerous studies from the health sciences that rely on randomized controlled trials have found

that reminders have a positive impact on engaging in various health-related behaviors (e.g.,

Armstrong, et al., 2009; Fry & Neff, 2009; Hurling, et al., 2007; Lantz, et al., 1995; Liang, et al.,

2011; Patrick, et al., 2009; Pop-Eleches, et al., 2011; Stockwell, et al., 2012). Most importantly

for our purposes, there is consistently strong empirical support that reminders have been found to

help individuals attend their scheduled healthcare appointments (McLean, Gee, Booth, Salway,

Nancarrow, Cobb, & Bhanbhro, 2014). Although utilized less frequently in the area of education,

a few recent randomized controlled trials (e.g., Castelman & Page, 2015, 2016) have found that

reminders are an effective means of getting students to follow through on the pre-matriculation

(e.g., registering for orientation and completing housing forms) and post-matriculation (e.g., re-

http://economicspsychologypolicy.blogspot.com/


filing the Free Application for Federal Student Aid) tasks that in turn lead to higher college

enrollment and retention rates, respectively.

Goal Framing

Levin, Schneider, and Gaeth (1998) describe goal framing as the manipulation of the

choice environment of a decision in order to “to focus attention on its potential to provide a

benefit or gain (positive frame) or on its potential to prevent or avoid a loss (negative frame)” (p.

167). The question of research studies that focus on goal framing is which characterization of the

goal will be more likely to persuade the decision-maker to adopt a specific behavior. Although

this is an empirical question that is specific to the particular behavior that is in question, framing

the decision as a potential loss of benefits is often found to be a more effective strategy. For

example, the use of a negative frame has been found to increase positive engagement among

individuals receiving temporary assistance for needy families (Farrell, Smith, Reardon, & Obara,

2016), to increase productivity among factory workers (Hossain & List, 2009), and to increase

voluntary compliance by taxpayers (Hasseldine, Hite, James, & Toumi, 2007). Use of a negative

frame is often effective because it exploits the fact that individuals are loss averse, and therefore

make choices that show a preference for avoiding a loss over making an equivalent gain

(Kahneman & Tversky, 1984). We will test for both types of framing in our research.

Study 1

Treatments

In this study we wanted to test whether the use of a reminder with or without goal

framing provided the week of the national test date could help fee waiver students remember and

be more committed to attend on their test date. We designed three different treatments for this

study, and the specific wording for each treatment is provided in Appendix A. Students in the


first treatment group received Email A, which included a statement that simply reminds students

of the test date and of the materials that they are required to bring to the test center. Students in

the second treatment group received Email B, which contained the same reminder as in Email A

with an additional statement framed to focus attention on the potential of their test day

attendance to provide a benefit. The additional statement read: “We believe that you can benefit

greatly from taking the ACT. When you take the ACT you are more likely to be approached by

colleges you may wish to attend.” Students in the third treatment group received Email C, which

contained the reminder plus a similar statement framed instead to focus attention on the potential

of their test day attendance to prevent a loss. This additional statement read: “DON’T MISS

OUT! If you do not attend, you might miss out on attending the college of your choice.”

The three different treatments were delivered on Wednesday, December 9, 2015, just

three days before the students’ test date. The mode of delivery for the treatment was email, as

texting students is currently prohibited under ACT policy. The rates at which students opened the

emails were consistent across the three treatments, at around 40% (see Table 1).

Population and Sample

The population for Study 1 was 169,534 students who used a fee waiver to register for

the December 12, 2015 ACT national test date. To ensure that we had sufficient statistical power

to detect a treatment effect when compared to the control group, we selected 40,000 students

from this population, using stratified random sampling to assign 10,000 fee waiver students each

to one of three treatment groups or the control group while preserving the representation of the

population by grade level, self-reported income, parents’ education level, and degree aspirations.

Descriptive statistics for the four groups are provided in Table 2. Given the timing of the

test date in late fall, the clear majority (81%) of fee waiver students in the sample were twelfth


graders. As expected among students receiving fee waivers, about 30% of these students reported

a family income of $24,000 or less and roughly half reported an income of $50,000 or less. Just

over a third of the sample were prospective first-generation college students, while about one

quarter of the sample had at least one parent with some college education below the bachelor’s

degree. Students in the sample had relatively high degree aspirations, with only 6% planning to

earn less than a bachelor’s degree. Roughly 60% of the sample were female, and about two-

thirds of the students were African American or Hispanic.

Analysis

Study 1 was conducted as a randomized controlled trial. We chose an intention-to-treat

(ITT) approach to the analysis, meaning that we kept all students in their assigned treatment or

controlled group regardless of whether they received the intended treatment (for the three

treatment groups) or unintentionally received the treatment due to contamination (for the three

treatment groups or the control group). For example, only about 40% of the students in each

treatment group actually opened the email. With the ITT approach, we include all treatment

group subjects in the analysis regardless of whether they opened the email. This changes the

interpretation of our findings to focus on the effectiveness of the treatment policy as opposed to

the effectiveness of the treatment as delivered.

The outcome examined in the current study—test-day attendance—is dichotomous,

where:

𝑦𝑖 = {1 𝑖𝑠 𝑝𝑟𝑒𝑠𝑒𝑛𝑡 𝑜𝑛 𝑡𝑒𝑠𝑡 𝑑𝑎𝑦

0 𝑖𝑠 𝑎𝑏𝑠𝑒𝑛𝑡 𝑜𝑛 𝑡𝑒𝑠𝑡 𝑑𝑎𝑦


Given the dichotomous nature of our outcome, we estimated a binary logit regression

model to test for differences in the likelihood of test-day attendance given the student’s

assignment to one of the treatment groups or the control group. This model takes the form:

ln (𝑝𝑖

1 − 𝑝𝑖) = 𝛽0 + 𝛽1𝑡1 + 𝛽2𝑡2 + 𝛽3𝑡3 + 𝜀𝑖

where pi is the probability of fee waiver student i attending on his or her test day, t1 is an

indicator variable for Treatment Group 1 (i.e, reminder only), t2 is an indicator variable for

Treatment Group 2 (i.e, reminder plus positive framing), and t3 is an indicator variable for

Treatment Group 3 (i.e, reminder plus negative framing); β0 is the estimated log odds of

attending for students in the Control Group, whereas β1, β2, and β3 are the estimated difference in

log odds between the respective treatment groups and the Control Group.

Results

Among the descriptive findings, approximately 75% of the fee waiver students in the

sample attended the December, 2015 ACT national test date. Descriptive differences in the test-

day attendance rates across the three treatment groups and the control groups were in the

hypothesized direction but were negligible in magnitude. The largest difference in attendance

rates—at roughly one-half percentage point—was between the reminder-only treatment group

(74.68%) and the control group (74.20%). The omnibus test and the parameter estimates for the

logistic regression model were not statistically significant, suggesting that none of the treatment

policies had the intended effect over the status quo. Table 3 provides the complete model results.

Study 2

Treatments

In this study we wanted to test whether the timing, frequency and content of reminders

that focus on common barriers to test-day attendance could help students to remove the possible


constraints that would make it easier for them to choose to attend on their test date. Findings

from an unpublished survey of ACT fee waiver students suggests that three particular constraints

are more prevalent among these students: 1) a prior work obligation, 2) caring for a family

member, and 3) a lack of transportation. We designed six treatments that allowed for variability

in the content and timing and frequency of the treatment (see Table 4 for design matrix).

Treatments 1-3 received the same content (Email D), which comprised a reminder of the test date

and required test materials as well as three “Quick Tips” for addressing the three potential

constraints. Treatments 4-6 had variations in the content (Emails E, F, and G) such that only one

of the three “Quick Tips” were provided with each reminder email. Specifically, we wanted to

test whether reducing the cognitive load on the students by focusing on a separate barrier each

week would be more effective at increasing attendance. The specific wording for each treatment

is provided in Appendix B. Treatment groups 1 and 4 received the first email three weeks prior

to the test date, with follow-ups sent two weeks prior to and the week of the test date. Treatment

groups 2 and 5 received the first email two weeks prior to the test date, with a follow-up sent the

week of the test date. Treatment groups 3 and 6 received only one email the week of the test

date. The mode of delivery for the treatment was email, as texting students is currently

prohibited under ACT policy. The rates at which students opened the emails varied between 32%

and 39%, with the open rates appearing to increase closer to the test date (see Table 5).

Population and Sample

The population for Study 2 was 72,062 students who used a fee waiver to register for the

June 11, 2016 ACT national test date. This population was much smaller than that for the

December, 2015 test date, and there are several possible explanations for this difference. First,

given the timing of admissions decisions at many colleges, very few twelfth graders have a need


to register for the June ACT national test date; this leaves only eleventh graders in the pool of

possible registrants. Second, given that many high school students are no longer in school during

the month of June, registration for this particular ACT test date tends to be lower than other test

dates. Finally, given that many eleventh graders are now taking the ACT in April through a

statewide in-school administration, fewer of these students are electing to register for the June

national test date two months later.

To ensure that we had sufficient statistical power to detect a treatment effect when

compared to the control group, we selected 70,000 students from this population, using stratified

random sampling to assign 10,000 fee waiver students each to one of six treatment groups or the

control group while preserving the representation of the population by grade level, gender,

underrepresented minority status (i.e., African American, American Indian, Hispanic, or Pacific

Islander), graduate degree aspirations, prospective first-generation college student status, and

self-reported high school grade point average.

Descriptive statistics for the seven groups are provided in Table 6. Given the timing of

the test date in late spring, the clear majority (86%) of fee waiver students in the sample were

eleventh graders. Roughly 63% of the sample were female, and about two-thirds of the students

were members of an underrepresented race/ethnicity. Just over a third of the sample were

prospective first-generation college students, while about one quarter of the sample had at least

one parent with some college education below the bachelor’s degree. Students in the sample had

relatively high degree aspirations, with only about 5% planning to earn less than a bachelor’s

degree and with over one-third of the students planning to earn a graduate degree. There was a

fairly good distribution of students across the high school grade point average categories.


Analysis

As in Study 1, Study 2 was conducted as a randomized controlled trial with an intention-

to-treat (ITT) approach to the analysis. This changes the interpretation of our findings to focus on

the effectiveness of the treatment policy as opposed to the effectiveness of the treatment as

delivered. See the analysis section for Study 1 for more details.

The outcome examined in the current study—test-day attendance—is dichotomous,

where:

𝑦𝑖 = {1 𝑖𝑠 𝑝𝑟𝑒𝑠𝑒𝑛𝑡 𝑜𝑛 𝑡𝑒𝑠𝑡 𝑑𝑎𝑦

0 𝑖𝑠 𝑎𝑏𝑠𝑒𝑛𝑡 𝑜𝑛 𝑡𝑒𝑠𝑡 𝑑𝑎𝑦

Given the dichotomous nature of our outcome, we estimated a binary logit regression

model to test for differences in the likelihood of test-day attendance given the student’s

assignment to one of the treatment groups or the control group. This model takes the form:

ln (𝑝𝑖

1 − 𝑝𝑖) = 𝛽0 + 𝛽1𝑡1 + 𝛽2𝑡2 + 𝛽3𝑡3 + 𝛽4𝑡4 + 𝛽5𝑡5 + 𝛽6𝑡6 + 𝜀𝑖

where pi is the probability of fee waiver student i attending on his or her test day, t1 through t6 are

indicator variables for treatment groups 1 through 6, respectively; β0 is the estimated log odds of

attending for students in the Control Group, whereas β1 through β6 are the estimated difference in

log odds between the respective treatment groups and the Control Group.

Results

Overall, approximately 73% of the fee waiver students in the sample attended the June,

2016 ACT national test date. Descriptive differences in the test-day attendance rates across the

six treatment groups and the control groups were mixed in direction and negligible in magnitude.

The largest positive difference, roughly 1.1percentage point, was between the fourth treatment


group (73.48%)—those who received three separate singularly-focused Quick-Tip email over the

three weeks—and the control group (72.37%). The largest negative difference, approximately -

0.9 percentage point, was between the sixth treatment group (71.48%)—those who received only

the transportation Quick-tip email during the week of the test—and the control group. The

omnibus test and the parameter estimates for the logistic regression model were not statistically

significant, suggesting that none of the treatment policies had the intended effect over the status

quo. Table 7 provides the complete model results.

Discussion and Future Directions for Research

Taking a college entrance exam such as the ACT or SAT can be an important step in the

college-going process. Despite the benefits of taking a college entrance exam, a large and

disproportionate share of students who register for the exam with a fee waiver do not

subsequently attend on their registered test date. To address this discrepancy, ACT created it’s

“Dialing for Scholars” research initiative with the goal of increasing the test-taking rate for

students who registered with fee waivers by reaching out to them in the weeks before their

scheduled national ACT test dates to proactively share information that would better prepare

them for the testing experience and encourage them to attend.

In the two studies discussed in this manuscript, we designed nine variations on an

intervention that would remind fee waiver students of their test date and “nudge” them to attend.

Despite having ample statistical power, we found that none of the nine interventions tested across

the two studies had a statistically significant impact on the attendance rates of treated fee waiver

students when compared to fee waiver students in their respective control groups.

There are several possible reasons for why our treatments were not effective, and those

reasons point to a future opportunity for research. One potential reason for the lack of positive


findings for our interventions is that they perhaps suffered from a poor mode of delivery for this

particular target population. Although these students provided an email address at the time that

they registered for the ACT, the email message open rates were quite low: 40% to 41% for the

December test date were, whereas the message open rates for the June test date (when many

students were already out of high school for summer break) ranged from 32% to 39%. A recent

survey of college undergraduates (Ha, Youngnya Joa, Gabay, & Kim, 2016) found that only 12%

of students used email as their primary mode of communication. Roughly 50% of the students,

however reported that text messaging on a cell phone was their primary mode of communication,

followed by social media (at 35%). Although some researchers (Castleman & Page, 2015, 2016)

are finding positive effects of nudges delivered by text messaging, texting students is currently

prohibited under ACT policy. In lieu of text messaging as a viable option, an alternative mode of

delivery for our treatments is needed.

Another possible reason why our treatments were not effective is that a low-touch nudge

from a testing company, regardless of mode, may simply not be enough to compel the students to

action. ACT, Inc. currently lacks the means to establish the kind of interpersonal connection (or

even familiarity) with the students that may be necessary to have a more meaningful dialogue

about the benefits of attending on test day, or to help the students address particular barriers to

test-day attendance, and this lack of connection may have contributed to the students feeling

little or no obligation to follow through on their commitment to attend on their test day.

If extra motivation or assistance in overcoming particular constraints to test day

attendance truly are what fee waiver students need, then perhaps a high-touch intervention from

a recognized authority figure would better serve the students’ needs. To this end, we are currently

designing a randomized controlled trial whereby we incentivize high school staff to reach out to


their fee waiver students in order to offer personal encouragement and to address the specific

test-day conflicts—such as work and family obligations and a lack of transportation—being

faced by their fee waiver students.


References

Allen, J. (2013). Updating the ACT College Readiness Benchmarks. ACT Research Report

Series 2013-6. Iowa City, IA: ACT, Inc.

Allen, J., & Sconing, J. (2005). Using ACT Assessment Scores to Set Benchmarks for College

Readiness. ACT Research Report Series 2005-3. Iowa City, IA: ACT, Inc.

Armstrong A. W., Watson, A. J., Makredes, M., Frangos, J. E., Kimball, A. B., & Kvedar, J. C.

(2009). Text-message reminders to improve sunscreen use: A randomized, controlled trial

using electronic monitoring. Archives of Dermatology, 145, 1230-1236.

Balfanz, R., & Byrnes, V. (2012). Chronic Absenteeism: Summarizing What We Know From

Nationally Available Data. Baltimore: Johns Hopkins University Center for Social

Organization of Schools.

Castleman, B. L., & Page, L. C. (2015). Summer nudging: Can personalized text messages and

peer mentor outreach increase college going among low-income high school graduates?

Journal of Economic Behavior & Organization, 115, 144-160.

Castleman, B. L., & Page, L. C. (2016). Freshman year financial aid nudges: An experiment to

increase FAFSA renewal and college persistence. The Journal of Human Resources, 51,

614-645.

College Board. (2016). The College and Career Readiness Benchmarks for the SAT Suite of

Assessments. K-12 Educator Brief. New York, NY: Author.

Criden, M. (2008). The Stranded Poor: Recognizing the Importance of Public Transportation for

Low-Income Households. Issue Brief. Washington, DC: National Association for State

Community Services Programs.


Farrell, Mary, Jared Smith, Leigh Reardon, and Emmi Obara (2016). Framing the Message:

Using Behavioral Economics to Engage TANF Recipients. OPRE Report 2016-02.

Washington, DC: Office of Planning, Research and Evaluation, Administration for

Children and Families, U.S. Department of Health and Human Services.

Fry, J. P., & Neff, R. A. (2009). Periodic prompts and reminders in health promotion and health

behavior interventions: Systematic review. Journal of Medical Internet Research, 11(2):

e16.

Ha, L., Youngnya Joa, C., Gabay, I., & Kim, K. (2016, April). Does U.S. College Students’

Social Media Use Affect School E-Mail Avoidance and Campus Involvement? Paper

presented at the annual conference of the Information and Telecommunications Education

and Research Association, Louisville, KY.

Hasseldine, J., Hite, P., James, S., & Toumi, M. (2007). Persuasive communications: Tax

compliance enforcement strategies for sole proprietors. Contemporary Accounting

Research, 24, 171–94.

Hurling R., Catt, M., De Boni, M., Fairley, B., Hurst, T., Murray, P., Richardson, A., & Sodhi, J.

(2007). Using internet and mobile phone technology to deliver an automated physical

activity program: Randomized controlled trial. Journal of Medical Internet Research,

9(2), e7.

Hossain, T., & List, J. A. (2009). The Behavioralist Visits the Factory: Increasing Productivity

Using Simple Framing Manipulations. NBER Working Paper No. 1562.

Kahneman, D., & Tversky, A. (1984). Choices, values, and frames. American Psychologist, 39

(4), 341–350.


Lantz, P. M., Stencil, D., Lippert, M. T., Beversdorf, S., Jaros, L., & Remington, P. L. (1995).

Breast and cervical cancer screening in a low-income managed care sample: The efficacy

of physician letters and phone calls. American Journal of Public Health, 85, 834-836.

Liang, X., Wang, Q., Yang, X., Cao, J., Chen, J., Mo, X., Huang, J., Wang, L. & Gu, D. (2011).

Effect of mobile phone intervention for diabetes on glycemic control: A meta-analysis.

Diabetic Medicine, 28, 455–463.

McLean, M. G., Booth, A., Salway, S., Nancarrow, S., Cobb, M. & Bhanbhro, S. (2014).

Targeting the Use of Reminders and Notifications for Uptake by Populations (TURNUP):

A systematic review and evidence synthesis. Health Services and Delivery Research, No.

2.34.

Patrick, K., Raab, F., Adams, M., Dillon, L., Zabinski, M., Rock C., Griswold W., & Norman G.

(2009). A text message-based intervention for weight loss: Randomized controlled trial.

Journal of Medical Internet Research, 11(1), e1.

Pop-Eleches, C., Thirumurthy, H., Habyarimana, J. P., Zivin, J. G., Goldstein, M. P., de Walque,

D., & Bangsberg, D. R. (2011). Mobile phone technologies improve adherence to

antiretroviral treatment in a resource-limited setting: A randomized controlled trial of text

message reminders. AIDS, 25(6), 825–834.

Samson, A. (Ed.) (2015). The Behavioral Economics Guide 2015 (with an introduction by Dan

Ariely). Retrieved from http://www.behavioraleconomics.com.

Samson, A. (2015). Behavioral Science: Theory and Practice. In A. Samson (Ed.), The

Behavioral Economics Guide 2016 (with an introduction by Dan Ariely) (pp. 1-27).

Retrieved from http://www.behavioraleconomics.com.


Stockwell , M. S., Kharbanda, E. O., Martinez, R. A., Vargas, C. Y., Vawdrey, D. K., &

Camargo, S. (2012). Effect of a text messaging intervention on influenza vaccination in

an urban, low-income pediatric and adolescent population: A randomized controlled trial.

The Journal of the American Medical Association, 307, 1702-1708.

Sunstein, C. R. (2014). Nudging: A very short guide. Journal of Consumer Policy, 37(4), 583-

588.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving Decisions about Health, Wealth, and

Happiness. Yale University Press: New Haven, NJ.

Wyatt, J., Kobrin, J., Wiley, A., Camara, W., & Proestler, N. (2011). SAT Benchmarks:

Development of a College Readiness Benchmark and its Relationship to Secondary and

Postsecondary School Performance. College Board Research Report 2011-5. New York,

NY: College Board.


Footnotes

1 Both organizations offer some alternative criteria for determining financial eligibility.

For more information, please see their respective websites at act.org and collegeboard.org.

2 Currently, ACT, Inc. policy does not allow the company to contact students via text

message.


Tables

Table 1

Email Open Rates for Study 1

Date Sent Segment Delivered Unique Opened Open Rate

12/09/15 Email A, Treatment 1 9957 4046 40.6

12/09/15 Email B, Treatment 2 9954 4011 40.3

12/09/15 Email C, Treatment 3 9948 3978 40.0


Table 2

Descriptive Statistics for Study 1

Variable T1 T2 T3 C1

Grade level* 11th 18.8 18.8 18.8 18.8

12th 81.2 81.2 81.2 81.2

Degree aspirations* Less than bachelor's 6.2 6.2 6.2 6.2

Bachelor's degree 46.7 46.7 46.7 46.7

Graduate degree 30.9 30.9 30.9 30.9

Missing 16.2 16.2 16.2 16.2

Self-reported Income* Less than $24k 29.8 29.8 29.8 29.8

$24k-$36k 19.5 19.5 19.5 19.5

$36k-$50k 11.7 11.7 11.7 11.7

$50k-$80k 9.1 9.1 9.1 9.1

$80k+ 3.6 3.6 3.6 3.6

Missing 26.3 26.3 26.3 26.3

Parents’ education* No college 35.3 35.3 35.3 35.3

Some college 26.4 26.4 26.4 26.4

Bachelor's degree 12.0 12.0 12.0 12.0

Graduate degree 4.9 4.9 4.9 4.9

Missing 21.3 21.3 21.3 21.3

Gender Male 39.2 39.7 38.6 39.2

Female 60.8 60.3 61.4 60.8

Race/Ethnicity African American 33.3 33.4 33.4 33.6

American Indian 1.3 1.4 1.3 1.3

Asian 4.4 4.2 4.1 4.3

Hispanic 31.7 31.7 32.0 32.4

Pacific Islander 0.3 0.3 0.3 0.4

Other race 7.6 7.9 7.7 7.5

White 21.4 21.1 21.2 20.6

Attendance rate

Attended 74.7 74.6 74.5 74.2

Absent 24.3 25.4 25.5 25.8

N 10,000 10,000 10,000 10,000

* Denotes variables used to stratify samples


Table 3

Logistic Regression Results for Study 1

Test DF Chi-Square Pr > ChiSq

Likelihood Ratio 3 0.710 0.871

Parameter Estimate Error Chi-Square Pr > ChiSq

Intercept 1.056 0.023 2136.177 0.001

T1 0.025 0.032 0.607 0.436

T2 0.022 0.032 0.443 0.506

T3 0.014 0.032 0.193 0.661

N=40,000


Table 4

Design Matrix for Study 2

Date T1 T2 T3 T4 T5 T6 C1

05/25/16 Email D Email E

06/01/16 Email D Email D Email F Email F

06/08/16 Email D Email D Email D Email G Email G Email G


Table 5

Email Open Rates for Study 2

Date Sent Segment Delivered Unique Opened Open Rate

05/25/16 Email D, Treatment 1 9664 3065 31.7

05/25/16 Email E, Treatment 4 9649 3190 33.1



06/01/16 Email F, Treatment 4 9643 3093 32.1

06/01/16 Email F, Treatment 5 9683 3504 36.2




06/08/16 Email G, Treatment 4 9643 3387 35.1




Table 6

Descriptive Statistics for Study 2

Variable T1 T2 T3 T4 T5 T6 C1

Grade level* 11th 85.7 85.7 85.7 85.7 85.7 85.7 85.7

12th 14.3 14.3 14.3 14.3 14.3 14.3 14.3

Gender* Male 37.3 37.3 37.3 37.3 37.3 37.3 37.3

Female 62.7 62.7 62.7 62.7 62.7 62.7 62.7

Underrepresented minority status*† Yes 66.0 66.0 66.0 66.0 66.0 66.0 66.0

No 34.0 34.0 34.0 34.0 34.0 34.0 34.0

Race/Ethnicity African American 25.9 25.7 26.5 26.1 25.8 26.2 25.7

American Indian 0.6 0.6 0.6 0.6 0.6 0.7 0.7

Asian 7.3 7.4 7.5 7.6 7.5 7.2 7.5

Hispanic 39.0 39.4 38.5 38.9 39.2 38.8 39.4

Pacific Islander 0.4 0.4 0.4 0.3 0.4 0.4 0.3

Other race 7.9 8.0 7.4 7.6 7.4 7.9 7.7

White 18.8 18.5 19.0 18.7 19.1 18.8 18.8

Graduate degree aspirations* Yes 37.4 37.4 37.4 37.4 37.4 37.4 37.4

No 62.6 62.6 62.6 62.6 62.6 62.6 62.6

Degree aspirations Less than bachelor's 5.0 5.2 5.0 5.0 5.0 4.9 5.0

Bachelor's degree 43.9 43.8 43.8 43.5 43.9 43.3 44.0

Graduate degree 37.4 37.4 37.4 37.4 37.4 37.4 37.4

Missing 13.8 13.6 13.9 14.1 13.8 14.4 13.7

Prospective first-generation student* Yes 36.5 36.5 36.5 36.5 36.5 36.5 36.5

No 63.5 63.5 63.5 63.5 63.5 63.5 63.5

Parents’ education No college 36.5 36.5 36.5 36.5 36.5 36.5 36.5

Some college 26.4 26.3 26.3 26.0 26.8 26.5 26.6

Bachelor's degree 13.3 13.5 13.1 13.3 12.9 12.8 13.1

Graduate degree 6.0 5.7 5.6 6.2 5.6 5.9 5.9

Missing 17.7 18.0 18.5 18.1 18.2 18.3 17.9


Table 6 (continued)

Variable T1 T2 T3 T4 T5 T6 C1

Self-reported HSGPA* 4.00 8.6 8.6 8.6 8.6 8.6 8.6 8.6

3.75-3.99 13.0 13.0 13.0 13.0 13.0 13.0 13.0

3.50-3.74 15.8 15.8 15.8 15.8 15.8 15.8 15.8

3.25-3.49 15.0 15.0 15.0 15.0 15.0 15.0 15.0

3.00-3.24 17.5 17.5 17.5 17.5 17.5 17.5 17.5

2.50-2.99 18.7 18.7 18.7 18.7 18.7 18.7 18.7

Less than 2.50 11.4 11.4 11.4 11.4 11.4 11.4 11.4

Attendance rate

Attended 72.6 73.2 73.1 73.5 72.5 71.5 72.4

Absent 27.4 26.8 26.9 26.5 27.5 28.5 27.6

N 10,000 10,000 10,000 10,000 10,000 10,000 10,000


Table 7

Logistic Regression Results for Study 2

Test DF Chi-Square Pr > ChiSq

Likelihood Ratio 6 13.712 0.033

Parameter Estimate Error Chi-Square Pr >ChiSq

Intercept 0.963 0.022 1853.927 0.000

T1 0.010 0.032 0.100 0.752

T2 0.044 0.032 1.911 0.167

T3 0.039 0.032 1.496 0.221

T4 0.056 0.032 3.120 0.077

T5 0.006 0.032 0.030 0.862

T6 -0.044 0.032 1.918 0.166

N=70,000


Appendix A

Treatments for Study 1

Email A

From: ACT Student Service <[email protected]>

Subject: IMPORTANT: Your ACT Test on Dec. 12

Dear [Student Name]:

This is an email to remind you that your ACT® test is on December 12, 2015.

What you need to do: Please report to your assigned test center by 8:00 a.m. on December 12

and be sure to bring a copy of your ticket and an acceptable photo identification with you, such

as a current official government or school photo ID. See www.actstudent.org for details.

If you have any questions about your test day, you can contact us at 319.337.1270 or email us at

[email protected].

Good luck on your test!

Yours truly,

[ACT Employee Name]


Email B





We believe that you can benefit greatly from taking the ACT. When you take the ACT you

are more likely to be approached by colleges you may wish to attend.

What you need to do: Please report to your assigned test center by 8:00 a.m. on December 12




[email protected].


Yours truly,

[ACT Employee Name]


Email C





DON’T MISS OUT! If you do not attend, you might miss out on attending the college of

your choice.

What you need to do: Please report to your assigned test center by 8:00 a.m. on December 12,




[email protected].


Yours truly,

[ACT Employee Name]


Appendix B

Treatments for Study 2

Email D

From: [ACT Employee Name], ACT <[email protected]>

Subject: Countdown to the ACT: [##] days to go!

Hi [Student Name],

Looks like you've registered for the upcoming June 11 ACT® test date with a fee waiver. This

means that ACT has covered the cost of registration for you because we know that taking a test

like the ACT is an important step in preparing you for life after high school. We really want you

to be present on June 11 to take advantage of this opportunity.

Quick Tips to Prepare for Your Test Day

Do you work on Saturdays? If so, talk to your employer now about taking off June 11.

Do you help care for a family member on weekends? If so, talk with other relatives and

care providers now about other arrangements for June 11.

Do you have a ride to your ACT test center? If not, talk to staff members at your school

now to help you arrange for transportation on June 11.


[email protected].


Yours truly,

[ACT Employee Name]


Email E


Subject: Countdown to the ACT: 17 days to go!

Hi [Student Name],





Quick Tip for the Week of May 23

Do you work on Saturdays? If so, talk to your employer now about taking off June 11.


[email protected].


Yours truly,

[ACT Employee Name]


Email F



Hi [Student Name],





Quick Tip for the Week of May 30

Do you help care for a family member on weekends? If so, talk with other relatives and

care providers now about other arrangements for June 11.


[email protected].


Yours truly,

[ACT Employee Name]


Email G



Hi [Student Name],





Quick Tip for the Week of June 6

Do you have a ride to your ACT test center? If not, talk to staff members at your school

now to help you arrange for transportation on June 11.


[email protected].


Yours truly,

[ACT Employee Name]

Documents

Nudging to No Effect: Attempts to Improve the College ... 2017 AEFP paper.pdf · Nudging to No Effect: Attempts to Improve the College Entrance Exam Attendance Rates of Low-Income