Sept. 6, 2019

Secondary and Postsecondary Indicators

Perkins V Methodology for Setting Performance LevelsPresenters:Tabatha Ries-Miller, Minnesota StateKari-Ann Ediger, Minnesota Department of Education

2

Introduction

3

• “State-determined levels of performance”

• Set by the State

– consultation and public comment

• All 4-years set in the 2020 state plan

• Requires “meaningful progress” yearly

• Set using “valid and reliable data”

• Revisions before 3rd year

• Revise at the end of any year if unanticipated circumstances, or changes in data or approaches

• Still required to meet 90%

Perkins V Transformation

4

• WIOA /Alternative Regression Model

– Isolate “unique impact” of program

– Time consuming –data collection and processing

– Complicated and difficult to explain

• Trend Extrapolation

– Presumes continued “normal” growth – business as usual

• Statistical Significance (Recommended)

– Evaluates if differences are large enough that we can conclude they are not just random variation

Exploring Methodologies

5

What is the relationship between state and local performance levels?

At what rate should we expect to see improvement?

How much of an increase is “meaningful progress” and also reasonably achievable?

Standard Deviation Method: Factors to Consider

6

4-year State Performance Level Methodology

7

Historical trend data

Cohort 2011-2013 2012-2014 2013-2015 2014-2016 2015-2017 2016-2018

Actual 45.9% 46.4% 47.4% 49.3% 49.7% 49.9%

Historical test

2013-2015(Baseline)

2017-2019Perf. Level

2017-2019Actual

2017-2019 90% Required

47.4% 50.7% 50.1% 45.6%

How much to increase? Exploring Historical Data for Reasonability

Standard Deviation = 1.6% X 2 = 3.3%

Example 2P1 (Earned postsecondary credential)

8

Baseline: 13.9%

Standard Deviation: 0.4%

4-year level: 0.8% increase over baseline

2024 Level: 14.7%

Baseline: 50.1%

Standard Deviation: 1.4%

4-year level: 2.8% increase over baseline

2024 Level: 52.9%

Baseline: 90.8%

Standard Deviation: 1%

4-year level: 2% increase over baseline

2024 Level: 92.9%

1P1: Postsecondary Retention and Placement

2P1: Earned Recognized Postsecondary Credential

3P1: Nontraditional Program Enrollment

Post-Secondary Indicators: Two Standard Deviations

9

12.5%

25%

50%

0%

25%

50%

75%

100%

0%

40%

70%

90%100%

Baseline Year 1 Year 2 Year 3 Year 4

Perc

ent

of

targ

et

Conservative Linear Aggressive

Three options:1. Conservative start with

more growth in grant year 3 and 4

2. Linear growth – equal growth each year

3. Aggressive start with more growth in grant year 1 and 2

Recommend: Conservative Start

How fast to grow? Exploring Trend Options

• Postsecondary cohort model – data lag

• Any change takes time to realize

• We can adjust methodology after 2nd year if needed before

Possible adjustment

10

All 4-year performance levels are set using two standard deviations. The standard deviation was calculated using simulated rates for 2012-2017 entry cohorts.

Recommended Postsecondary State Levels of Performance

Indicator Baseline Year

BaselineRate

4-year Increase

Year 1 Year 2 Year 3 Year 4

1P1: Postsecondary Retention and Placement

2016-2018

90.85% 2.05% 91.10% 91.36% 91.87% 92.90%

2P1: Earned Recognized Postsecondary Credential

2017-2019

50.06% 2.80% 50.41% 50.76% 51.47% 52.87%

3P1: Nontraditional Program Enrollment

2017-2019

13.87% 0.82% 13.98% 14.08% 14.28% 14.70%

11

The 4-year performance level for 1S1 and 1S2 are set using a two standard deviation increase over the baseline, while 2S1 and 2S2 was set using a one standard deviation increase. The standard deviation for 1S1 was based on 2011-2018 data, while 1S2, 2S1, and 2S2 was based on 2015-2018 data.

Recommended Secondary State Levels of Performance

Indicator Baseline Year

BaselineRate

4-year Increase

Year 1 Year 2 Year 3 Year 4

1S1: Four-year Graduation Rate

2018 91.50% 2.42% 91.80% 92.11% 92.71% 93.93%

1S2: Extended-yearGraduation Rate

2018 93.06% 1.70% 93.27% 93.48% 93.91% 94.75%

2S1: Academic Proficiency: Reading/Language Arts

2018 56.06% 2.35% 56.35% 56.65% 57.23% 58.41%

2S2: Academic Proficiency: Mathematics

2018 45.25% 1.19% 45.40% 45.55% 45.84% 46.44%

12

The 4-year performance level for 4S1 is set using a two standard deviation increase over the baseline, 5S2 was set using a one standard deviation increase and 5S4 was set using a ½ standard deviation. The standard deviation for 4S1, 5S2, and 5S4 were based on 2015-2018 data.

Recommended Secondary State Levels of Performance

Indicator Baseline Year

BaselineRate

4-year Increase

Year 1 Year 2 Year 3 Year 4

3S1: Post-Program Placement

TBD

4S1: Non-traditional Program Concentration

2018 37.57% 1.57 % 37.77% 37.96% 38.35% 39.14%

5S2: Program Quality: Postsecondary Credit

2018 38.94% 2.35% 39.23% 39.53% 40.11% 41.29%

5S4: Program Quality: TSA/IRC

2018 67.35% 2.60% 67.68% 68.00% 68.65% 69.95%

13

Questions or Feedback?

State Levels of Performance

14

Local Level of Performance Methodology

15

• Meaningful progress every year

• Aggregate must at least equal state performance level

• Take into account differences among locals

• Reasonable based on historical differences

• Acknowledge that not all locals will reach state determined performance level

– Some local’s 2017 performance was lower than 2013, so they would not have met 0% increase

Local Methodological Parameters

16

Local Levels: Equal to Opportunity to Improve

Successful Outcomes

State’s Total Concentrators

Opportunity to Improve State’s Total

Opportunity to Improve

Each Consortium’s Opportunity

State’s 4-year Increase (E.g. 2%

Increase)

Consortium’s 4-year increase

17

Measure State Baseline

Recommended Methodology

State 4-year Increase

State 4-year Perf. Level

Ranges of Local Rate Increases

Range of Local N Increases

Estimate of % Locals Success (90%)

1P1: Postsecondary Retention and Placement

90.85% 2.05% 92.90% 0.84% to 6.42%

0 to 26 97%

2P1: Earned Recognized Postsecondary Credential

50.06% 2.80% 52.87% 1.20% to 4.28%

0 to 49 83%

3P1: Nontraditional Program Enrollment

13.87% 0.82% 14.70% 0.66% to 0.89%

0 to 11 83%

How do the numbers shake out?Postsecondary Indicators

18

Measure State Baseline

Recommended Methodology

State 4-year Increase

State 4-year Perf. Level

Ranges of Local Rate Increases

Range of Local N Increases

Estimate of % Locals Success (90%)

1S1: Four-year Graduation Rate

91.50% 2.42% 93.93% 0.73% to 5.18%

1 to 89 100%

1S2: Extended-yearGraduation Rate

93.06% 1.70% 94.75% 0.82% to 5.20%

3 to 73 100%

2S1: Academic Proficiency: Reading/Language Arts

56.06% 2.35% 58.41% 1.75% to 3.41%

2 to 75 62%

2S2: Academic Proficiency: Mathematics

45.25% 1.19% 46.44% 0.97% to 1.64%

1 to 38 73%

How do the numbers shake out?Secondary Indicators

19

Measure State Baseline

Recommended Methodology

State 4-year Increase

State 4-year Perf. Level

Ranges of Local Rate Increases

Range of Local N Increases

Estimate of % Locals Success (90%)

3S1: Post-Program Placement

TBD

4S1: Non-traditional Program Concentration

37.57% 1.57 % 39.14% 1.17% to 2.34%

2 to 52 88%

5S2: Program Quality: Postsecondary Credit

38.94% 2.35% 41.29% 1.29% to 3.83%

3 to 125

88%

5S4: Program Quality: TSA/IRC

67.35% 2.60% 69.95% 0% to 6.97%

0 to 25 62%

How do the numbers shake out?Secondary Indicators

20

Questions or Feedback?

Local Levels of Performance

21

Secondary

• Kari-Ann Ediger

Kari-Ann.Ediger@state.mn.us

651-582-8269

Postsecondary

• Tabatha Ries-Miller

Tabatha.Ries-Miller@minnstate.edu

651-201-1848

Contact Us with Comments or Questions

22

MINNESOTA STATE AND MINNESOTA DEPARTMENT OF EDUCATION ARE AFFIRMATIVE ACTION, EQUAL OPPORTUNITY EMPLOYERS AND EDUCATORS.

30 East 7th Street, Suite 350St. Paul, MN 55101-7804

www.minnstate.edu/system/cte

1500 Highway 36 WestRoseville, MN 55113

www.education.state.mn.us/MDE/dse/cte