Embed Size (px)
Citation preview
Multivariate Models of Student Success
RP Group/CISOA ConferenceApril 28, 2009
Granlibakken, Lake Tahoe
Dr. Matt WetsteinDean of Planning, Research and Institutional Effectiveness
San Joaquin Delta College
Alyssa Nguyen Bri HaysResearch Analyst
Research Analyst
Introduction
Many studies of student success rely on
quasi‐experimental designs
Intervention is tested for its effect, usually
with a control group
Downside – lack of multiple control variables
More and more, researchers are turning to
multivariate logistic regression models
Introduction
Logistic Regression’s appeal
Many of our dependent variables of interest are well suited for dichotomous analysis
Techniques have become standard in packages like SAS, STATA, SPSS
Allows for multivariate analysis and more holistic understanding of student behavior
Introduction
RP Group researchers are leading the way in
recent years – some examples…
Wurtz (2008) Logit model for generating placement test recommendations
Spurling (2007) Logit model of prior English enrollment on GE course success
Younglove (2009) Logit model to recommend concurrent course enrollment for basic skills students
CSS (2002) Logit model to validate prerequisites for enrollment in nursing programs
Introduction
Some notes on Logit
S‐shaped curve
Should be little collinearity among independent variables
Goodness of Fit
Reliable models have non‐significant Chi Square values using the Hosmer‐Lemeshow goodness of fit test
Model’s ability to correctly classify cases vs. modal guessing strategy
My prior use of Logit –
explaining judicial voting behavior in the U.S. & Canadian Supreme Court
Model of Student Success
Current interest: developing multivariate
models to examine patterns of student success
Background traits (SES, ethnicity, income)
Skill levels
Norms & attitudes toward college & transfer
Engagement in college life & services
Model of Student Success
Course takingpatterns &unit loads
Assessment scores
GPA history
Orientationcounselingclubstutoring
Student’s Background Characteristics
George Kuh et. al. 2005. Student Success in College:
Creating Conditions that Matter.
San Francisco: Jossey
Bass.
Colleen Moore & Nancy Shulock. 2007. Beyond the Open
Door: Increasing Student Success in the California Community
Colleges. Sacramento: CSU Sacramento
Steve Spurling. 2007. The Impact of an Attained English
Competence on Subsequent Course Success. Journal of
Applied Research in the Community College, 15 (1): 29‐36.
Vince Tinto. 2008. Student Success and the Building of
Involving Educational Communities, in Promoting Student
Success in College,
http://soeweb.syr.edu/academics/grad/higher_education
Keith Wurtz. 2008. A Methodology for Generating
Placement Rules that Utilizes Logistic Regression. Journal
of Applied Research in the Community College 16 (1): 52‐58.
Student Demographics – Delta College
30,111 students in 2007‐08
58% female
11% African American
28% Hispanic/Latino
20% Asian/Pacific Islander
Average age is 24.8
45% qualify for fee waivers (income guideline
or child of disabled vet/deceased vet)
Student Demographics – Delta College
African Americans are underrepresented
when examining AA degree attainment, transfer status, and completion of “critical
4”
courses (1. ENG 1A, 2. COMST 1A, 3. ENG 1B/1D/PHIL 30, and 4. Transfer MATH)
Hispanics lag behind other groups on several
measures (transfer success, degrees, critical 4 attainment)
Models of Student Success
Multivariate Models –
Cohort Used: Fall 2007 Students
Predictor Variables
BACKGROUND
ENGAGEMENT
Age
Number of Counseling Services
Gender (1 = Female, 0 = male)
Tutoring Hours (Math/Science)
Ethnic Group (1 = White, 0 = Non‐White)
DSPS Status (1 = DSPS)
EOPS Status (1 = EOPS)
Low income (1 = BOG fee waiver)
NORMS/TRANSFER DIRECTED
SKILLS
Student Education Plan (1 = yes)
Reading Assessment Level (1, 2, 3)
Units Attempted
Math Assessment Level (1, 2, 3)
Prior course work in ENG 79/1A
Math GPA (where relevant)(0 = No courses, 1 = success in ENG 79, 2 = success in ENG 1A, 3 = success in both)
Models of Student Success
Dependent Variables
Success in Large Enrollment/Gateway Courses (Success defined as
grade of A, B, or C)
Psychology 1
History 17A
Political Science 1
Math 82 (Transfer Algebra)
Persistence to Spring 2008 term
Fall 2007 Overall Success Rate(Success defined as Semester GPA >= 2.00)
All dependent variables 1 or 0, with positive outcome = 1
Predicted success = a + bx1 + bx2 + bx3 …e
Prior success
in English is
the
strongest
predictor of
success in
PSYCH 1
Problem –
coefficients
don’t have
same
meaning as
in OLS
Regression
Table 1 – Predictors of Student Success in Introduction to Psychology (PSYCH 1) Using a Logistic Regression Model (Fall 2007)
Two‐tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term .022 .072 68.6 81.5 12.9 White Student .227 .174 68.8 73.4 4.6 Female Student .069 .645 69.2 70.7 1.5 DSPS Student .731 .223 69.8 82.8 13.0 EOPS Student .226 .445 69.7 74.3 4.6 BOG Fee Waiver ‐.275 .091 72.2 66.4 ‐5.8 Skill Levels Math Level .315 .002 ** 64.5 77.3 12.8 Reading Level ‐.016 .898 70.5 69.8 ‐0.7 Norms/Seriousness Prior English Success .543 .000 *** 58.7 87.9 29.2 Attempted Units .072 .001 *** 52.0 74.8 22.8 Educational Plan ‐.477 .047 * 71.3 60.7 ‐10.6 Engagement Counseling Services .067 .022 * 67.7 85.1 17.4 Orientation Class .360 .053 68.5 75.7 7.2 Constant ‐1.691 .000 Hosmer Lemeshow Test 5.08 .749 Nagelkerke R Square .171 % Correctly Classified 69.0% Reduced Error Measure 5.2% N 1,038 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level
Making it relevant
Logit coefficients need to be converted to meaningful data
Step 1 – Set x to a particular value (example, prior English success = 0)
Step 2 – Calculate the equation z score using mean * coefficient for other variables
Step 3 – Compute the antilog of the equation result
Step 4 – Compute the odds of success by using the formula: antilog/(1+antilog) or EXP/(1+EXP)
Holding all other variables constant, the result tells you the odds of success with no prior English success (ranges
between 0.0 & 1.0)
Making it relevant
N = 1,038
Predicted Probability of Success in Psychology 1 and English Course Taking Patterns
58.771.0
80.887.9
0.0
20.0
40.0
60.0
80.0
100.0
No English Completed BelowTransfer English
CompletedTransfer English
Two EnglishCourses
Pred
icted Odd
s of Suc
cess
Age and
prior
success in
English
are the
strongest
predictors
of success
in HIST
17A.
Table 2 – Predictors of Student Success in U.S. History (HIST 17A) Using a Logistic Regression Model (Fall 2007)
Two‐tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term .044 .002 ** 38.4 71.9 33.5 White Student .277 .071 39.6 46.4 6.8 Female Student .141 .360 40.4 43.8 3.4 DSPS Student 1.323 .070 42.0 73.1 31.1 EOPS Student .216 .509 41.9 47.3 5.4 BOG Fee Waiver ‐.269 .129 44.4 37.9 ‐6.5 Skill Levels Math Level .219 .026 * 37.7 48.4 10.7 Reading Level .253 .040 * 40.7 53.2 12.5 Norms/Seriousness Prior English Success .372 .000 *** 31.4 58.3 26.9 Attempted Units .027 .200 35.4 44.4 9.0 Educational Plan ‐.268 .207 43.4 37.0 ‐6.4 Engagement Counseling Services ‐.034 .210 43.5 31.6 ‐11.9 Orientation Class .284 .160 41.1 48.1 7.0 Constant ‐2.871 .000 Hosmer Lemeshow Test 3.81 .874 Nagelkerke R Square .124 % Correctly Classified 64.4% Reduced Error Measure 24.6% N 841 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level
Making it relevant
N = 841
Predicted Probability of Success in U.S. History and English Course Taking Patterns
31.439.9
49.158.3
0
20
40
60
80
100
No English Completed BelowTransfer English
CompletedTransfer English
Two EnglishCourses
Pred
icted Odd
s of Suc
cess
Table 3 – Predictors of Student Success in U.S. Government (POLSC 1) Using a Logistic Regression Model (Fall 2007)
Two‐tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term .008 .590 62.6 68.4 5.8 White Student .089 .603 62.6 64.6 2.0 Female Student .117 .451 61.8 64.6 2.8 DSPS Student 1.096 .112 62.8 83.5 20.7 EOPS Student ‐.005 .987 63.3 63.2 ‐0.1 BOG Fee Waiver ‐.157 .375 64.5 60.9 ‐3.6 Skill Levels Math Level .297 .006 ** 57.6 71.1 13.5 Reading Level .115 .392 60.6 65.9 5.3 Norms/Seriousness Prior English Success .343 .000 *** 50.3 73.9 23.6 Attempted Units .041 .049 * 51.9 65.7 13.8 Educational Plan ‐.106 .619 63.8 61.3 ‐2.5 Engagement Counseling Services .038 .189 61.7 74.0 12.3 Orientation Class .639 .006 ** 61.0 74.8 13.8 Constant ‐2.871 .000 Hosmer Lemeshow Test 9.862 .275 Nagelkerke R Square .115 % Correctly Classified 65.4% Reduced Error Measure 9.8% N 821 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level
Prior
success in
English is
the
strongest
predictor of
success in
POLSC 1.
Making it relevant
N = 821
Predicted Probability of Success in POLSC 1 and English Course Taking Patterns
50.358.8
66.873.9
0
20
40
60
80
100
No English Completed BelowTransfer English
CompletedTransfer English
Two EnglishCourses
Pred
icted Odd
s of Suc
cess
Prior
success
in Math is
the
strongest
predictor
of
success
in
IntermedAlgebra
Note the
impact of
tutoring
Table 4 – Predictors of Student Success in Intermediate Algebra (MATH 82) Using a Logistic Regression Model (Fall 2007)
Two‐tailed Odds When Odds When Change Variable Coefficient p value X is Low X is High in Odds Background Age at term .025 .039 * 48.0 67.3 19.3 White Student ‐.168 .348 54.6 50.4 ‐4.2 Female Student .002 .992 51.8 51.9 0.1 EOPS Student ‐.492 .273 52.3 40.2 ‐12.1 BOG Fee Waiver ‐.186 .302 53.5 48.9 ‐4.6 Skill Levels Math Level .026 .809 51.0 52.3 1.3 Prior Math GPA .454 .000 *** 27.9 70.4 42.5 Norms/Seriousness Attempted Units .026 .247 44.7 53.8 9.1 Educational Plan .082 .688 51.4 53.4 2.0 Engagement Tutoring Hours .018 .068 51.4 66.1 14.7 Orientation Class ‐.297 .168 53.2 45.8 ‐7.4 Constant Hosmer Lemeshow Test 10.04 .262 Nagelkerke R Square .127 % Correctly Classified 63.3% Reduced Error Measure 24.3% N 701 *** Significant at 99.9% confidence level, ** significant at 99% level, * significant at 95% level
Making it relevant
N = 701
Predicted Probability of Success in Intermediate Algebra and Prior Success in Math Classes
37.8
48.9
60.170.4
0
20
40
60
80
100
Prior Math GPA =1.0
Prior Math GPA =2.0
Prior Math GPA =3.0
Prior Math GPA =4.0
Pred
icted Odd
s of Suc
cess
Term to Term Persistence
A number of variables helped explain persistence, including key indicators of engagement (i.e., counseling & orientation services)
All other things being equal, the more counseling services received, the greater the likelihood of student persistence
N = 11,060 students
Overall Term GPA (2.0 or higher)
N = 11,060 students
Holding all other variables constant, greater amounts of counseling produce greater odds of term GPA exceeding 2.0. The
same applied for higher reading and math assessment levels, being a woman, being older, and taking more units.
Uses of the Data
GE course success presented to Social
Science faculty
Response – Transfer English advisory on all division GE courses
Orientation data presented to counselors &
matriculation committee
Response – Student services departments are exploring new modes of orientation to make it more universal
Uses of the Data
Learning Center data presented to Title V
Steering Committee, Learning Center Directors
Data will be presented at HACU Conference in Fall 2009
PSYCH 1 STUDENTS FALL 2007 Z Score Z Score EXP/(1+EXP) EXP/(1+EXP) Change in
Variable b Mean b*mean X Values Low High EXP Low EXP High Odds Low Odds High Odds
age 0.022 21.40 0.471 18 v. 50 0.780 1.484 2.181 4.410 0.686 0.815 13.0%
gender 0.069 0.64 0.044 0 v. 1 fem 0.811 0.880 2.250 2.410 0.692 0.707 1.5%
ethnic 0.227 0.29 0.066 0 v. 1 white 0.789 1.016 2.201 2.762 0.688 0.734 4.7%
reading ‐0.016 1.97 ‐0.032 1 v. 3 0.870 0.838 2.387 2.312 0.705 0.698 ‐0.7%
math 0.315 1.82 0.573 1 v. 3 0.596 1.226 1.816 3.409 0.645 0.773 12.8%
orientation 0.360 0.22 0.079 0 v. 1 yes 0.775 1.135 2.172 3.113 0.685 0.757 7.2%
sep ‐0.477 0.12 ‐0.057 0 v. 1 yes 0.912 0.435 2.489 1.545 0.713 0.607 ‐10.6%
eops 0.226 0.09 0.020 0 v. 1 yes 0.834 1.060 2.303 2.887 0.697 0.743 4.5%
bog ‐0.275 0.37 ‐0.102 0 v. 1 yes 0.956 0.681 2.602 1.977 0.722 0.664 ‐5.8%
dsps 0.731 0.02 0.015 0 v. 1 yes 0.840 1.571 2.317 4.812 0.698 0.828 12.9%
counseling 0.067 1.74 0.117 0 v. 15 0.738 1.743 2.092 5.715 0.677 0.851 17.5%
prior english 0.543 0.93 0.504 0 v. 3 0.351 1.980 1.420 7.242 0.587 0.879 29.2%
units attmpt 0.072 11.78 0.848 1 v. 15 0.079 1.087 1.082 2.964 0.520 0.748 22.8%
constant ‐1.691 1.00 ‐1.691
zscore 0.855
Predicted Probability 0.894 2.444747 0.7097
No English 58.7 1.4369418 4.207808 0.80798
Completed Below 71.0
Completed Trans 80.8
Two English Cour 87.9 Predicted Probability of Success in Psychology 1 and English Course Taking Patterns
58.771.0
80.887.9
0.0
20.0
40.0
60.0
80.0
100.0
No English Completed BelowTransfer English
Completed TransferEnglish
Two EnglishCourses
Pred
icted Odd
s of Suc
cess
