Our goal for this presentation

THE FROYO EFFECT: USING SMALL INCENTIVES TO

ACHIEVE A 51% UNIVERSITY-WIDE RESPONSE

RATE

Office of Institutional Research and AnalysisDirector, Janel Sutkus, Ph.D.

Research Designer and Analyst, Hyun Kyoung Ro, Ph.D.

Our goal for this presentation

To describe – in detail – how we administered a small incentive program for our spring census survey that resulted in a 51% response rate (a response rate we had not achieved since Spring 2005)

Outline

Study backgroundResponse rate history and goalSurvey schedule by cohortsThe big RQSurvey populations (on-campus and off-campus)Marketing planAdministration

CalendarInvitation and incentive messages

OutcomesResponse rates and odds of responding Redemption of small incentives

Questions and discussion

Study background

The Carnegie Mellon University-Student Experience Study (CMU-SES) is a university-wide study designed to understand: how well, if at all, we are meeting our educational objectives for students which experiences, resources, and activities affect those educational outcomes when they do affect the outcomes, in what direction and magnitude The CMU-SES was designed to replace the End of Year Survey which was administered to undergraduates in 2000, 2001, 2004, 2005, and 2009. It will be administered to all CMU students on a three-year cycle, with the next administrations in the spring semesters of 2015 and 2018.

The core planning team began meeting on June 1, 2011 with these working principles: this will not be a revision of past EOY’s, but an entirely new study the development process will be research question-driven, rather than survey item-driven, and the instrument will be created hand-in-hand with the analysis plan (we will hypothesize that certain predictors affect particular outcomes, and will test those hypotheses using regression methods) satisfaction is not one of our primary educational objective, so this will not be a satisfaction survey we cannot study everything we want to know, so we will set our priorities and study a few things in a thorough manner

We spent ten months creating and testing this survey.

Survey and assessment schedule 11-12 to 18-19Year 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19

F S F S F S F S F S F S F S F SCohort

Fall 2011CIRP SES ADV ADV SES

Fall 2012 ADV ADV SES ADV BCSSE NSSE NSSE

Fall 2013 ADV SES ADV ADV

Fall 2014 CIRP SES ADV ADV SES

Fall 2015 ADV ADV SES ADV BCSSE NSSE NSSE

Comparisons: Comparisons: Comparisons:FY CIRP SES Fall CIRP to Spring SES BCSSE NSSE /ADV Fall BCSSE to Spring NSSE ADV FY ADV to JR ADVSoph ADV FY SES to SR SES ADV FY NSSE to SR NSSE SES JR ADV to SR ADVJunior ADV Soph ADV to JR ADV SES FY ADV to Soph ADV ADVSenior SES NSSE/ADV Soph ADV to SR ADV ADV

CIRP: Cooperative Institutional Research Program Freshman Survey (UCLA-HERI) Spring 2012 SESBCSSE: Beginning College Student Survey of Engagement (Indiana University) Spring 2013 ADV NSSE (FY and senior only)NSSE: National Survey of Students Engagement (Indiana University) Spring 2014 ADVSES: Carnegie Mellon University Student Experience Study (CMU IR&A) Spring 2015 SESADV: Carnegie Mellon University Undergraduate Advising Survey (CMU IR&A) Spring 2016 ADV NSSE (FY and senior only)

Spring 2017 ADVSpring 2018 SESSpring 2019 ADV NSSE (FY and senior only)

Response rate history for spring census surveys

UG pop randomly split in two

April 2000 End of Year





April 2009 Advising

April 2011 Advising

39.7%

32.3%

58.6%

53.0%

32.7% 31.6% 33.0%

The big goal and associated RQ

By using a series of small incentives, can we return our spring census response rate to the levels we had in the mid-2000’s?

We hedged our bets by also providing a large “raffle-type” incentive, but our plan was to do analysis at the end to determine whether small incentives could replace large incentives in the future. RQ: Are there differences in the odds of responding during a particular small incentive window for students who are eligible for small incentives compared to students who are not eligible for small incentives?

Survey population

ON CAMPUS: 5517 students Were eligible for a drawing for one of two iPad3’s (or $500 at Bookstore) AND received a small incentive for completing the survey

OFF CAMPUS: 117 students on study abroad were eligible for a drawing for one of two iPad3’s (or $500 at Bookstore)

LIMITATION AND CAVEAT: Other than the big difference of being off-campus and having study abroad experience, the two groups were much the same in terms of demographics and ultimate survey responses

Off campus students were not aware what they were missing

Marketing plan

“Who says? CMU-SES!” branding campaign

Banner in University Center

Table tents at dining locations

Flyers

“Tabling” on the Cut with balloons and branded candy

Core team had CMU-SES t-shirts with iron-on transfers

Who says? CMU-SES

Student

Experience

Study

Administration calendarSunday Monday Tuesday Wednesday Thursday Friday Saturday

25 26 27 28 29 SURVEY LAUNCH11:00 PM

30 LAUNCH EVENT AT UC4-6 PM

31

1 2GIVEAWAY #1 ENDS AT 10 PMREMINDER #1 SENT AT 10 PM

3 GIVEAWAY #1AT UC 3-6 PM(FroYo)

4 5 6 7

8 9 10 11GIVEAWAY #2 ENDS AT 10 PM

12 GIVEAWAY #2AT UC 3-5 PM(Fitness class card)REMINDER #2SENT AT 10 PM

13 14

15 16 17 18GIVEAWAY #3 ENDS AT 10 PMREMINDER #3SENT AT 10 PM

19 GIVEAWAY #3AT CARNIVAL MIDWAY3:30-5 PM(Funnel cake)

20 21

22 23 24 25GIVEAWAY #4 ENDS AT 10 PMREMINDER #4SENT AT 10 PM

26GIVEAWAY #4AT UC 3-5 PM(Maggie Murph coffee)FINAL REMINDERSENT AT 10 PM

27 SURVEY CLOSES 2 PMWRAP EVENT AT UC5-7 PM(CMU gear and iPad3 drawing)

28

Dear Andrew,

We are writing to invite your participation in the Carnegie Mellon University Student Experience Study (CMU-SES). Every three years the university conducts a study of the entire student body to gain insight into the educational experience at CMU. Your input is extremely valuable and should take fewer than 10 minutes using the link below. Students who complete the survey will be placed into a drawing for one of two iPad3’s (the recipients may choose to substitute $500 of merchandise at the University Stores). In addition, all students who complete the survey between now and 10:00 PM on Monday, April 2, will be eligible to pick up a certificate for a free FroYo at Skibo Café.

Please click here to participate in the survey: <<link>>Or copy and paste the URL below into your internet browser: <<URL>>

This study is sponsored by the Vice Provost for Education and the Dean of Student Affairs and is administered by the Office of Institutional Research and Analysis. Your individual responses will be kept confidential and will only be reported in the aggregate. No information that could identify your responses will be reported. The survey will be open until Friday, April 27, at 2:00 PM. Thank you for your participation. If you have any questions about the study, please contact Janel Sutkus, Director of Institutional Research and Analysis, at [email protected]. Sincerely,Amy Burkert, Vice Provost for EducationGina Casalegno, Dean of Student Affairs

Invitation

Not included for students

studying abroad

End of survey messages

On campus:

Thank you for completing the CMU-SES!

Watch your e-mail for details on how to claim your thank you gift and for your invitation to the closing party on April 27 where we will announce the winners of the two iPad3's (you do NOT need to attend to win).

Studying abroad:

Thank you for completing the CMU-SES!

You have been entered into the drawing for one of two iPad3's.

Dear Andrew,

Thank you for participating in the CMU-Student Experience Study. This e-mail is your ticket for TWO things:

1) You have won a certificate for a free FroYo at the Skibo Café. Please pick up your certificate at:The University Center, Wean Commons on Tuesday, April 3, from 3:00 to 6:00 PM.

You will need your student ID and this e-mail to receive your certificate. You can either print this e-mail or show it on your smartphone. The certificate will be valid until May 11.

2) You are invited to the CMU-SES closing party at:The University Center, Connan Room on Friday, April 27, from 5:00 to 7:00 PM.The party will include refreshments and music, and students who attend will receive tickets for drawings for CMU gear.We will also announce the winners of the two iPad3's (you do not need to attend to win).You will need your student ID and this e-mail to attend. You can either print this e-mail or show it on your smartphone.

We look forward to seeing you there.

Janel Sutkus and Hyunny Ro Amy Burkert Gina Casalegno and Holly HippensteelOffice of Institutional Research and Analysis Office of the Vice Provost for Education Office of the Dean of Student Affairs

E-mail upon completion

Certificates

This certificate is good for one FroYo at the Skibo Café.

Thanks for participating in CMU-SES!

# Expires 5/11/12

This certificate is good for one funnel cake at the A Phi O Booth at Carnival.


# Expires 4/21/12

This certificate is good for one Group X fitness class (includes yoga classes) at the University Center.


# Expires 5/4/12

This certificate is good for one 16 ounce brewed coffee or tea at Maggie Murph Café.


# Expires 5/11/12

Each certificate was hand-numbered in green ink and samples were given to each vendor.

3/29 3/30 3/31 4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 4/9 4/10 4/11 4/12 4/13 4/14 4/15 4/16 4/17 4/18 4/19 4/20 4/21 4/22 4/23 4/24 4/25 4/26 4/27

484

662

8648

354

257

41

199 163

44 4488

159

71

Reminder

Reminder

ReminderReminder

FinalReminder

Survey launched

at 11 PM

Responses by date

Responses by incentive

Invited R15635

Invited R24301

Invited R33669

Invited R43209

Invited R53104

Responded315

Responded105

Responded460

Responded632

Responded1334

Incentive FroYo Fitness class Funnel cake Coffee None

Response rate 23.7% 14.7% 12.5% 3.3% 10.1%

Odds of responding

89.8% higher odds

no difference in odds

688.3%higher odds



in the specific incentive window for a student in the small incentive population compared to a student not in the small incentive population, and controlling for gender, race, citizenship, and class year

Incentive redemption

Week Incentive Days active

Cost Responses N Collected

% Collected

Dist at Closing Party

Cost if all redeemed

N redeemed

% redeemed

Actual cost

1 FroYo 4 $3.00 1334 183 13.7% 20 $609 136 67.0% $408

2 Fitness Class 10 Gift 632 17 2.7% 5 29.4%

3 Funnel Cake 6 $3.50 460 76 16.5% $266 62 81.6% $217

4 Coffee 7 $1.25 105 7 6.7% 10 $21 12 70.6% $15

ALL 27 2531 283 11.2% 30 $947 215 76.0% $640

Total costs

Incentives Cost Marketing Cost Closing Party Cost

All small incentives $640 Banner $144 Balloons and decorations $146

Two iPad3’s $998 Balloons $25 Food (pizza, popcorn, cookies, hummus trays) $468

T-shirts, candy, supplies $122 Drinks $32

$1638 $291 $646

$2575

Compared to the most recent census survey (April 2011 Undergraduate Advising), we paid $143 per additional response percentage point or $2.61 per additional student

Future plans

CMU-SES in Spring 2015:GOAL: Maintain response rate in the low 50’sMarketing and branding continuesNo large incentivesKeep small incentives (replace fitness class and coffee)Anticipated budget - $1700

New “health behaviors” survey in Fall 2013:GOAL: Response rate in low 50’s to match with

FY’s and sophs from CMU-SES in Spring 2013Create marketing and branding campaignGuaranteed small incentives (FroYo, other tasty but healthy treats)Raffle every two days for Fitbugs

In the time since. . .

UG pop randomly split in two

39.7%

32.3%

58.6%

53.0%

32.7% 31.6% 33.0%

51.0%

33.4%

One large incentive: iPad Mini

For further information

[email protected]

[email protected]

Documents

Our goal for this presentation