A multi-institutional study of student success course impact

A Multi-Institutional Study of Student Success Course ImpactBeth Mulherrin, University of Maryland University College

Sandy Daston, PAR Framework

Figuring Out What Works For Student Success

Public university offering online degree programs to a diverse population of working adults

Largest open access public online university in U.S.

Premier provider of higher education to U.S. military since 1949

Part of the University System of Maryland

About UMUC

Jumpstart

Focus on Connection and Entry touchpoints (readiness and motivation, goal setting and commitment)

Problem-solving approach (diagnostics, tools for self-assessment, career goals)

Areas of strength and concern-academic and personal- to support goals and mitigate risks

CAPL 101

Setting the Stage

Online Learning Readiness

Jumpstart Results- RetentionOne Year Retention RatesFall 2013:

◦Successful JS Completers vs. Successful Matched: 96.8% versus 79.3%

Spring 2014:

◦All JS Participants vs. All Matched: 88.5% vs. 58.4%

◦Successful JS Completers vs. Successful Matched: 93.0% versus 57.1%

Jumpstart Results- Re-enrollmentSignificantly higher rates of re-enrollment into next consecutive term for:◦ Fall 2013:

◦ All JS vs. All Matched: 90.0% vs. 70.3%◦ Successful JS vs. Matched: 96.8% vs. 72.4%

◦ Spring 2014◦ All JS vs. All Matched: 68.3% vs. 38.2%◦ Successful JS vs. Matched: 73.2% vs. 36.5%

◦ Fall 2014 (New Students Only)◦ Successful JS vs. Matched: 78.9% vs. 66.1%

PAR ResearchCommon elements for positive first year retention outcomes with reviewed success courses include:

-academic planning/degree planning-information about program of study-connection with advising-study skills, time management-assessment of strengths + possible barriers to success

The Predictive Analytics Reporting (PAR) FrameworkLaunched as a research project at the Western Interstate Commission for Higher Education’s Cooperative for Educational Technologies.

Funded by Bill & Melinda Gates Foundation 2011, 2012, 2014

Supported by 6, then 16, then 22 forward-thinking institutions and managed by WCET to evaluate and test the use of predictive analytics to explore student loss and momentum.

Now a self-sufficient and Independent 501.c.3.

What PAR Does

PAR uses descriptive, inferential and predictive analyses to create benchmarks, institutional predictive models and to inventory, map and measure student success interventions that have direct positive impact on behaviors correlated with success.

PAR by the Numbers2.2 million students and 20 million courses in the PAR data warehouse, in a single federated data set, using common data definitions.

48 institutions

77 discrete variables are available for each student record in the data set. More than 2 dozen additional constructed variables used to explore specific dimensions and promising patterns of risk and retention.

Over 450 institution interventions organized by predictors they address and point in student life cycle.

Structured Readily Available Data

Common data definitions = reusable predictive models and meaningful comparisons.

Openly published via a cc license @ https://public.datacookbook.com/public/institutions/par

https://public.datacookbook.com/public/institutions/par

Data Inputs- Common Data Definitions

Student Demographics & Descriptive

GenderRace

Prior CreditsPerm Res Zip Code

HS InformationTransfer GPAStudent Type

Student Course Information

Course LocationSubject

Course NumberSection

Start/End DatesInitial/Final Grade

Delivery ModeInstructor Status

Course Credit

Student Academic Progress

Curent Major/CIPEarned Credential/CIP

Student Financial

InformationFAFSA on File – Date

Pell Received/Awarded –Date

Course CatalogSubject

Course NumberSubject LongCourse Title

Course DescriptionCredit Range

Lookup TablesCredential Types Offered

Course Enrollment PeriodsStudent Types

Instructor StatusDelivery Modes

Grade CodesInstitution Characteristics

Possible Additional Placement TestsNSC InformationSES Information

Satisfaction SurveysCollege Readiness Surveys

Intervention Measures

Intervention MeasurementUsual Analysis StepsFind a Reasonable Comparison Group

Ensure Data Understanding & “Completeness”

Define Outcome Measures◦ Term: GPA, >=C Credits Earned, Credit Ratio, Withdrawals, Credits Attempted

◦ Term to Term: Deltas

◦ Retention (if possible)

Iterations◦ Simple binary analysis- no models

◦ Apply models

©PAR Framework 2015

Intervention MeasurementChallengesFind a Reasonable Comparison Group

◦ Often no real controls

◦ Pre implementation vs. Post Implementation

◦ Participated vs. did not (optional interventions/course by course)

◦ Determine control variables

Ensure Data Understanding & Completeness


Intervention Measurement–Student Success Courses

ObjectivesDraw comparisons between program elements

◦ existing PAR Framework data set

◦ existing PAR access to information about student success courses

Compare cross institutional results

◦ using PAR available outcomes measures


Intervention Measurement –Student Success Courses

Step 1: Identify Participating PartnersOnline Survey to Partners (11 respondents)

◦ Student Success Course Offerings

◦ Academic Planning

◦ New Student Orientation

◦ College Readiness Assessment


Step 1: Identify Participating PartnersStudent Success Course/Academic Planning Offerings # of Respondents

Student Success Course Required for All Students 4

Student Success Course Required for Some Students 5

No Student Success Course Offering 2

Have Academic Planning/Degree Mapping Intervention

(varying % of students impacted)

10

No Academic Planning/Degree Mapping Interventions 1


Intervention Measurement –Student Success Courses

Intervention MeasurementStudent Success Courses

Step 1: Identify Participating PartnersAvailability of a comparison group

◦ Implemented after 2009 (0)

◦ Required for some, not all (5)



Step 1: Identify Participating PartnersSinclair Community College (1 credit)

University of Central Florida (3 credits)

University of North Dakota (2 credits)

Rio Salado Community College (1 or 3 credits)◦ 1- credit short course

◦ 3-credit in-depth course

Excelsior College (3 credits)◦ For Military/Veteran students

◦ For all other students



Step 2: Additional Info GatheredWho took the course? (required or strongly encouraged)

◦ Lack preparation/poor prior performance

◦ Special populations

◦ Poor performance at the institution

◦ Certain majors and/or undeclared

Course syllabus

Type of instructor teaching the course



Step 3- Analysis Qualitative

Quantitative



Step 3- Qualitative AnalysisDetermine Primary Course Components

Sources◦ Survey Responses

◦ Syllabus


Student Success Course Components # of courses

Academic Planning/Degree Mapping 8

Information about Programs of Study 8

Connection with Advising Services 8

Study Skills/Time Management 8

Social Engagement Opportunities 8

Assessment of Strengths and Possible Barriers to

Success8

Information about Academic Support Resources 7

Career Planning 7

Financial Aid/Money Management Information 6

Being an Engaged Learner/ Critical Thinking 6

Explicit Goal Setting 6

Student Success Course Components # of Courses

Information Literacy/Research Skills 5

Being a Student HERE (at this institution) 5

Being a College Student (various ways/expectations

for independence)5

Instruction on Technologies Used (Blackboard,

Google Drive, etc.)4

Health & Wellness 3

College Readiness Assessments 2

Community Service Project 1

Common Read 1

Peer Mentor Assigned to the Course 1

Intervention Measurement Student Success Courses

Step 3- Quantitative Analysis (Performed by individual institution)Determine Target & Comparison Groups

◦ Treatment Group: Students taking the Student Success Course, all PAR years

◦ Comparison Group: Students who did not take the Student Success Course


Step 3: Quantitative Analysis

Outcomes StudiedRetention: active course records between 12 months and 18

months after first course start date, or if they credentialed during or before that same period

Credit Ratio: # credits earned with a C or better grade in the student’s first 12 months divided by the student’s credits attempted in the same time period (binned into 2 categories around the institutional mean)



Identify Control Variables (Per institution)

• Credential Level Sought (Associates, Bachelors, Certificate)

• Development Education Participation

• Delivery Mode (Fully Online, Fully Onground, Mix of both)

• Pell Grant• Full/Part Time• GED • Military or Reservist

• Veteran• Race• Student Age• Prior Credits (Yes or No)• High School GPA (applicable only

for institutions with available data on HS GPA)*

• Gender• Student Type (per the institution’s

definitions)

* When used, students with no data were not included in the analysis



Control Variables - Details◦ For 1 institution: used additional info on intervention exposure

◦ For 1 institution: limited analysis to eliminate known extremely high risk students



Methods◦ Logistic Regression

◦ Effect of course participation was added after the significant control variables had been entered in the model as predictors

◦ Model selection was based on a balance of model fit and inclusion of appropriate control variables



Proportion of students taking vs not taking the student success course varied markedly, from a high of approximately 40% of the student population taking the course at one institution to a low of less than 1% of students at the other end of the range.


Step 3: Quantitative Analysis Results

(p values ranged from <.02 to <.0001, with most in the latter highly significant range)


Results Summary

Only one of the eight Student Success Courses analyzed showed a statistically significant positive effect on 12 month credit ratio for students taking the course vs. those who did not

For retention, six of seven courses showed a significantly positive effect on retention (1 course not in effect long enough to measure retention)


Course Component Summary


Course Component Summary


Interpretation & Next Steps

Important to consider factors not controlled for (other interventions, past performance)

Optional courses may attract more motivated students

More exploration on factors that make a difference/patterns

Faculty connection


Summary & Contact Info

[email protected]

[email protected]

mailto:[email protected]

mailto:[email protected]