EXCEL WORKSHOP: Teaming for Results

2016 NCCEP/GEAR UP CAPACITY-BUILDING WORKSHOP 1

SKILL-BUILDING SEMINAR:The College and Career

Readiness Dashboard


A PARTNERSHIP FOR PREDICTING STUDENT SUCCESS IN HIGHER EDUCATION


AGENDA

1. Introductions2. Overview of

Data Use by the University of Oklahoma in Student Retention K20 Center The College & Career Dashboard Partnership

3. Building the CCR Dashboard Model4. The CCR Dashboard 5. Q&A


UNIVERSITY COLLEGEThe UNIVERSITY of OKLAHOMA

2016 NCCEP/GEAR UP CAPACITY-BUILDING WORKSHOP


RETHINKING RETENTION atthe UNIVERSITY of OKLAHOMA


Internal research conducted by Doug Gaffin, OU, 2011.

Percent Retention

RETENTION LANDSCAPE

Presenter

Presentation Notes

The graph below depicts the retention “landscape” at OU, with purple peaks indicating the highest retention (i.e., 90-100%) and blue areas indicating the lowest retention (i.e., 60-70%). The white dashed line super imposed over the graph indicates the cut off for automatic admissions. When we look at the lower left-hand pocket, in the trough, just above the automatic admissions cut off of 50% rank and 24 ACT, we see a deep hole of student attrition; because they made the automatic cutoff, they were not admitted on contract and exhorted/induced to take advantage of academic support services. Having more discretion to judge the interaction of the three admission criteria (HS GPA, rank, and ACT score), especially in the context of the college-preparatory nature of a given high school empowers OU to admit better-equipped students, and to better equip the students we admit about whom we have concerns.

HIGH RISK

ACT Scores

Rank

Percent Retention

PREDICTING RETENTION: TRADITIONAL PREDICTORS

Low Risk: 90% total retention

Uncertain Risk

High Risk: Don’t meet acadreqs for auto admit

90% ret / ACT 28.8 / GPA 3.92 / 26.4% sample




ADDITIONAL PREDICTORS

• HS Academic Engagement• Class Size• Financial Concerns• Alumni Ties• Institutional Commitment

Stacked Model, Including Additional Risk Factors: Financial Concerns, Class Size, Alumni Ties, Academic Engagement, and Institutional Commitment

26.4% 34.3%

68.6% 51.2%

5.014.5%

10 15 20 25 30 35 20

Prob

abili

ty o

f Ret

entio

n

0.2

0.4

0.6

0.

8

1

.0

HS GPA, HS Academic Engagement, Class Size, Financial Concerns, ACT, Alumni Ties & Institutional Commitment

ACT Score


BETTER IDENTIFYING AND SERVING AT-RISK STUDENTS

Move to holistic admissions Modeling predicted retention with readily available,

objective data

PredRet2Pr(Herei) = B1(HS GPAi) + B2(ACTi) +

B3(HS Class Sizei) + B4(Sectori) + ei

PredRet1Pr(Herei) = B1(HS GPAi) +

B2(ACTi)+ ei

“PredRet3”Pr(Herei) = B1(HS GPAi) + B2(ACTi) +

B3(HS Class Sizei) + B4(Sectori) + B5(AppDatei) + ei

CONSIDERING ADDITIONAL PREDICTORS

Comparing the PredRet models:Predicted and actual retention

Predicted Retention Groupings

Actu

al R

eten

tion


USING DATA TO INCREASE RETENTION

Using data to predict retention Holistic admissions Wait-list decisions Para-retention committee

Using data to impact retention Cohort analysis Call back campaign Processes and procedures

Using unmet need to target talks during NSEP Application deadlines


THE K20 CENTER

2016 NCCEP/GEAR UP CAPACITY-BUILDING WORKSHOP

Whole School CCR

CultureTeacher’s Practice

Leaders

Student Readiness

THE COLLEGE AND CAREER READINESS MODEL


K20 GEAR UP PROJECTS

K20 Pathways to SUCCESS

FY2008 32 school districts 3,800+ students Cohort Graduated

in 2014

• K20 GEAR UP for the PROMISE

• FY2011• Oklahoma City

Schools• 5,000+ students• Cohort Graduates

in 2017 & 2018

THE COLLEGE AND CAREER READINESS DASHBOARD

Presenter

Presentation Notes

?>>


THE PROBLEM

SCHOOLS are inundated with data (attendance, GPA, OCCT, EOI, EXPLORE, PLAN,

ACT, PSAT, SAT, etc…) data are reported at grade level and delivered in formats (paper-based

and electronic reports) that limit educator use, and data remains in silos, are rarely aggregated, and are used as

summative evaluation tools.



THE PURPOSE

The PROMISE Dashboard will allow students, parents, and schools to explore how academic and behavioral choices along with their current performance level on selected benchmarks (e.g. ACT EPAS, GPA, Attendance, EOI Exams) projects to likelihood of success in their first year (and ongoing persistence) in an institution of higher education within the state of Oklahoma.


THE PARTNERSHIP

The University of Oklahoma P20 Data Council Oklahoma State Regents of Higher Education Oklahoma State Department of Education District Partners


PROCEED WITH CAUTION

Longitudinal Data Sets Cross Traditional Boundaries (district, state department, and higher education)

Privacy Concerns Research Oversight – IRB Accessibility


BUILDING THECOLLEGE & CAREER READINESS MODEL

Oklahoma


PREDICTIVE MODELING

The process by which models are created or chosen to best project outcomes.

Models are simply a mathematical or statistical combination of variables collected prior to the outcome being observed.

Six stages of the Predictive Modeling Process


SIX STAGES OF THE PREDICTIVE MODELING PROCESS 1. Problem Description X2. Understanding the Data X3. Preparing the Dataset X4. Creating the Models X5. Evaluating the Models X6. Implementing the Models


CURRENT PROGRESS

Data from OCPS, OSRHE, and OSDE have been obtained and useful datasets have been constructed.

We have matched and aggregated a longitudinal student record for all available 6-13 data fields .

We have created a smaller set of useful predictors and constructs for analysis within the larger variable set.


CURRENT PROGRESS

We have conducted analyses developing initial multi-level linear and logistic regression modeling to illuminate the relationships among the predictors and outcomes.

We have begun implementation of our data models that would allow a comparison of a student’s current and possible future performance against predicted performance in higher education using DASHBOARD technology.


MULTI-COHORT DESIGN


UNDERSTANDING THE DATA

After merging the two datasets: 19,728 students 32,435 variables Approximately 640 million data points Data dictionary runs 1179 pages Printing the dataset would require 1.26 million pages Large datasets are considered to be at least 10 million data points by

data mining advocates. We have 64 times that number!


DATA CLEANING

Initial Data Point Count: • 639,877,680

Clean Data Point Count by Grade:• 6th: 914,874• 7th: 1,178,094• 8th: 1,230,123• 9th: 1,382,274• 10th: 1,267,368• 11th: 1,028,403• 12th: 892,119


OUTCOME VARIABLES

Freshman GPA Persistence - 1st-year retention GPA > 2.00 Enrolling in a College Course All outcomes can be statistically-adjusted for variables that

influence outcomes external to the Grades 6-12 predictors. Type of College Coursework (e.g. STEM courses)


OUTCOME VARIABLES

Predictive modeling will use different outcome variables for different purposes

Fine-grained focused analyses are possible If I take a grade-level course in 9th grade and make an B, what

is my predicted first-year GPA if I went to OU/OSU? How does that compare to taking a below grade-level course

and making an A?


PREDICTOR VARIABLES AND LEVELS OF ANALYSIS

The nature of a set of predictor variables allows us to conceptualize the dataset into levels on which statistical modeling occurs

Level 0 – Outcome variable (adjustments) Level 1 – Person-level Level 2 – School - level Level 3 – Within-person change over time


PREDICTOR VARIABLES (Level 1)

Academic Variables Test Performance (e.g. EOI, OCCT, ACT, EXPLORE) GPA (e.g. Overall, Math/Science) Special Designations (e.g. Gifted, Spec Ed) Advanced Placement (E.g. #courses, AP Credit) Promotion/Retention Course-Taking Pattern



Enrollment/Attendance Information Entry and Exit Codes (Transfers, Expelled) Attendance (Total rate, Percentage Unexcused) Tardies

Demographic (Controls) English Language Proficiency US Resident Status Homeless Status Free/Reduced Lunch Status Sex and Ethnicity



Behavior (Disciplinary Issues) Number of referrals Type of Referrals Chronic Offending Actions Taken (e.g. suspensions) Days Suspended



Behavior (Academic Engagement) Academic planning for future ENGAGE data

Commitment Family Attitudes Homework Completion Study Skills Etc.



Behavior (PsychoSocial) ENGAGE

Social Connection Optimism Self-Confidence Goal Striving Determinism Etc.



School-level Data Average Demographics (e.g. Percent F/R lunch) Average Academic Performance (API, ACT, etc) Average Disciplinary levels (rate of suspensions) Student/Teacher Ratio ESL/ELL Ratio Special Ed Ratio Attendance Ratio Size of School Number of Courses offered Etc.


CREATING THE MODELSEXAMPLE MODELS OF INTEREST: Impact of math course taking patterns on first year college GPA. Impact of attendance, behavioral issues, and GPA on college enrollment.


9TH GRADE MATH COURSE TAKING MODEL

Augment the Level Zero adjusted first-year GPA model with 9th grade student-level predictors (Level 1)

MODEL: First Year College GPA = Constant + (w1*D_C + w2*Reg_Univ +

w3*State_Univ) + (b1*HSGPA + b2*Grade_Level_Math_Course + b3*Above_Grade_Level_Math_Course) + residual

BETA WEIGHTS WERE ESTIMATED AS:

LEVEL 0 LEVEL 1Constant = 1.09W1 = -1.99 B1 = 0.21W2 = -0.14 B2 = 0.10W3 = 0.08 B3 = 0.63


USING THE MODEL

A student achieves a 3.5 High School GPA and completes an above-level math course during his/her 9th grade year. What is his/her predicted first-year GPA at a large state school (OU or OSU)?

GPA = Constant + w1*D_C + w2*Reg_Univ + w3*State_Univ + b1*HSGPA + b2*Grade_ Level_Math_Course + b3*Above_Grade_ Level_Math_Course

GPA = 1.09 + (0)*D_C + (0)*Reg_Univ + (1)*State_Univ + (3.5)*HSGPA + (0)*GLMC + (1)*AGLMC

GPA = 1.09 + (0)*-1.99 + (0)*-0.14 + (1)*0.08 + (3.5)*0.37 + (0)*0.05 + (1)*0.38

GPA = 1.09 + 0 + 0 + 0.08 + 1.295 + 0 + 0.38 = 2.845


PREDICTION ERROR

When computing an predicted GPA for a student, we also compute a Standard Error of the Mean Predicted Value to create a possible range for the predicted GPA.

The 9th grade math course taking model has a Standard Error of the Mean Predicted Value = .08.

Therefore, the student’s predicted first year college GPA will range from 2.765 to 2.925.


PRESENTATION FORMAT

Rather than presenting a projected GPA and a margin of error, we will present the likelihood that a student will obtain a COLLEGE GPA falling in one of 4 criterion groups. These GPA groupings are: 2.0-2.49 2.5-2.99 3.0-3.49 3.5+


ESTIMATING THE LIKELIHOOD

The predicted value of a student’s first year, college GPA is acquired by plugging the student specific characteristics into the model and solving.

Once this is done a scale score (z-score) is computed by using the following equation:

𝑍𝑍 = 𝑀𝑀𝑀𝑀𝑀𝑀 𝐶𝐶𝐶𝐶𝑀𝑀𝐶𝐶𝐶𝐶𝐶𝐶𝑀𝑀𝐶𝐶𝑀𝑀 𝐺𝐺𝐺𝐺𝐺𝐺 −𝐺𝐺𝐶𝐶𝐶𝐶𝑃𝑃𝑀𝑀𝑃𝑃𝐶𝐶𝐶𝐶𝑃𝑃 𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆𝐶𝐶𝑆𝑆𝑀𝑀𝑃𝑃𝑆𝑆𝐶𝐶𝑃𝑃 𝐸𝐸𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶


CREATING PROBABILITIES

In this example, the Z-score is used to locate how much of the distribution exists beyond the specified point. This is equal to the probability of achieving this GPA.


MODELS IMPLEMENTED

ACT/SAT Model Model for students in grades 9-12. Makes predictions based on ACT or

SAT score and overall GPA. AP/IB Course Model

Model for students in grades 10-12. Makes predictions based on students’ enrollment in Advanced Placement and/or International Baccalaureate classes.

Behavior Referrals/Days Suspended Model Model for students in grades 6-12. Makes predictions based on the

number of behavioral referrals and number of days students have been suspended.


MODELS IMPLEMENTED

CRT Models Models for each Math, Reading, and Science for students in grades 6-

10. Makes predictions based on CRT scores. EOI Models

Models for Algebra I, English II, and Other End of Instruction tests. Algebra I model covers grades 6-12. English II and Other models cover grades 9-12. Makes predictions based on EOI scores.


MODELS IMPLEMENTED

Course-Taking Models Models that make predictions based on their grade level compared to

the grade level of math and sciences courses in which they enroll. Makes predictions for students in grades 6-12 based on whether their math or science courses are below, at, or above grade level.


MODELS IMPLEMENTED

Multiple Predictor Model A model for students in grades 6-12 that accounts for multiple

variables that include: Below/At/Above grade level math Overall GPA Math GPA ACT/SAT score Enrollment in Advanced Placement/International Baccalaureate

classes


IMPLEMENTING THE MODELS


EVALUATING THE MODELS

Validating the models K-fold cross-validation methodology Divide sample into k-groups Use 1 group to develop model Validate on k-1 other subgroups Iterate this process across the K groups

Validate using other school districts


THE K20 COLLEGE & CAREER READINESS DASHBOARDFocus Group


FOCUS GROUP

4 Counselors from Oklahoma City Public Schools 5+ years of experience as counselors Counselors from

1 High School 3 Mid-High School


FOCUS GROUP FEEDBACK: LIKES

• Immediate user feedback and readability of the graphs

• Dynamic nature of the graphs—they change as the data entered changed

• Data comes from OKCPS students and that the graphs compared colleges in a side-by-side manner


FOCUS GROUP FEEDBACK: REQUESTS

Adding… a GPA calculator to help students better understand how grades affect

GPA, model for the impact of concurrent enrollment on college success, and clearer definitions of categories of colleges and the categories of

students.


FOCUS GROUP FEEDBACK: HOW THEY WOULD USE THE DASHBOARD

Found it easily adaptable for all grade levels and it compliments other data aggregating systems

Most useful during pre-enrollment and in the transition from middle school to high school (8th to 9th grade) to help guide students’ course choice and during parent nights.


FOCUS GROUP FEEDBACK: HOW THEY WOULD USE THE DASHBOARD

Large schools would most likely use the dashboard in a large group classroom guidance scenario, and

Small schools could use this during individual guidance sessions.


THE K20 COLLEGE & CAREER READINESS DASHBOARDIn Action


http://ccr.k20center.ou.edu/




THE CCR DASHBOARD IN ACTIONUser Stories

My ACT score is back, now “what”?Meet James. He is a Junior who was recently contacted by a regional college basketball coach. As a sophomore, he sat for the ACT and scored a 14. He confesses he did not prepare for the test. James does enough in the classroom to stay off the ineligibility list and his GPA is a 2.0. How could use the CCR Dashboard to explore the impact of improving his academic credentials?

Presenter

Presentation Notes

Note to facilitator: Use the College Entrance Exam model. Choose 11th grade, then select ACT, and slide the bars to 14 and 2.0. Now, show the student what would happen in they increased their ACT by four points. Move the bar four points to 18 and keep GPA the same. Note the change. Now, move the bar back to 14 and show the student what would happen if they score the same on the ACT, but increase GPA to 3.0. Note the change. Now move the ACT bar to 18 and keep the GPA bar at 3.0 and show them the results after increasing both.

Why is “when”I take a course important?

Meet Barbara. She is one of many students in a 7th grade classroom who are deciding on what classes they should enroll in for their 8th grade year. Based upon her 7th grade math teacher’s recommendation, her enrollment form includes the option of Algebra I. Her best friend leans over and whispers, “Algebra I is hard, let’s save it for 9th

grade!” Let’s use the CCR Dashboard to explore the implications of this decision.

Presenter

Presentation Notes

Note to Facilitator: Use the MVP model. Choose 8th grade. Slide the GPA bar to 2.5 (average). Choose Pre-Algebra and slide the GPA bar to 2.5. Note the graphs. (There is no difference at the eighth grade level.) Now, go back to the calendar icon at the left and change the grade to 9th (since this decision ultimately impacts their 9th grade math choice as well.) Keep the GPA at 2.5. Now, choose Algebra I from the drop down menu. Note the graph. Change the graph to Geometry and note the change.

“When” is it too late for my GPA?Meet Claudia. She is starting her 10th

grade year. As Middle School student, she was an excellent student, carrying a 4.0. However as a freshman, she struggled with the transition and only managed a 1.8. She wants to go to college, but is worried her GPA is too low to ever be successful and is at the point of just giving up. Let’s use the CCR Dashboard to explore how improving her GPA impacts her college readiness.

Meet Andrae. He is an Junior and is a good student with a GPA of 3.4. He has not taken any AP courses because he heard the classes are challenging. Andrae is wondering if the extra work is worth it. How can you use the CCR Dashboard to show the student the impact taking an AP course could have on his college success?

Does “what” courses you take really impact college readiness?

Presenter

Presentation Notes

Note to Facilitator: Use the MVP model. Choose 8th grade. Slide the GPA bar to 2.5 (average). Choose Pre-Algebra and slide the GPA bar to 2.5. Note the graphs. (There is no difference at the eighth grade level.) Now, go back to the calendar icon at the left and change the grade to 9th (since this decision ultimately impacts their 9th grade math choice as well.) Keep the GPA at 2.5. Now, choose Algebra I from the drop down menu. Note the graph. Change the graph to Geometry and note the change.


NEXT STEPS

Continue to Develop Grade-Specific Models

Implement Level 2 and Level 3 models

Work with Interested Parties (e.g. IHEs)


THANK YOU

Dr. Nicole [email protected]

Dr. Leslie [email protected]

Dr. Robert [email protected]

Dr. Scott [email protected]

mailto:[email protected]




Documents

EXCEL WORKSHOP: Teaming for Results