KEVIN DOUGHERTY, DENISA GANDARA, NICK HILLMAN, ALICIA KINNE-CLAWSON, AMY LI, TREY MILLER,

TRAVIS REINDL

THE STATE OF PERFORMANCE FUNDING IN THE STATES: ORIGIN, DESIGN, IMPLEMENTATION, AND EFFECTS

KEVIN J. DOUGHERTY

ASSOCIATE PROF. OF HIGHER EDUCATION, TEACHERS COLLEGE, COLUMBIA UNIVERSITY

PERFORMANCE FUNDING FOR HIGHER EDUCATION: FORMS, EXTENT, AND ORIGINS

PF 1.0

• E.g. TN (1979), FL (PBB: 1996-2008), MO (2013)

• Bonus over and above base state funding. Typically between 1 and 5%

• Typical performance Indicators• Output: Graduation, Job placement, Licensure

exam passage• Intermediate achievement: Retention,

Developmental ed completion, Transfer

Source: Burke (2002); Davies (2014); Dougherty & Reddy (2013); NCSL (2014)

PF 2.0

• E.g. IN (2009), OH (2009), TN (2010)• Performance indicators embedded in base state

funding• Proportion of state appropriations can get quite

high: Up to 85-90% (TN)• More emphasis on intermediate indicators:

• Course completions (OH)• Completion of developmental educ. (OH, TN)• Reaching certain credit thresholds (OH, TN)

Source: Davies (2014); Dougherty & Reddy (2013); Dougherty & Natow (in press); NCSL (2014)

EXTENT

• 36 states have ever established a PF program• 28 to 30 states are operating a PF program now• Two waves of performance funding adoptions

• Wave 1 (1979-2000): 21 states initially adopting; 5 states re-adopting (adding to existing programs or re-establishing PF after having it lapse). Most were PF1

• Interregnum (2001-2006): 3 states initially re-adopting; 3 re-adopting

• Wave 2 (2007- ): 12 states initially adopting; 19 re-adopting. Half the programs are PF 2.0.

Source: Davies (2014); Dougherty & Natow (in press); NCSL (2014)

STATUS OF PF (HCM STRATEGISTS, 2014)DISAGREEMENTS REGARDING STATES AND

DEFINITIONS

WAVE 1 ORIGINS - SUPPORT

• Data: 7 states: FL, IL, MO, OH, SC, TN, WA (Dougherty & Natow, in press)

• Supporters• State elected officials (especially GOP legislators)• Business • State HE boards and Higher education institutions

• Supporters’ Motives• Need to increase efficiency of government and higher educ.

Market-oriented methods make government agencies more efficient

• Find new means of securing more funds for higher education in time of fiscal stringency.

WAVE 1 ORIGINS - OPPOSITION

• Opponents: • State universities: mostly passive resistance• Community colleges (SC only)

• Motives• PF provides state officials with excuse to cut back on

regular state funding of higher ed.• PF intrudes on autonomy of higher education

institutions.• PF programs fail to tailor performance indicators to

differing institutional missions and student bodies.

Wave 1 Origins – Facilitating conditions

• Policy learning• Internal sources• External sources: Other states, policy organizations,

feds, consultants

• Policy windows or external shocks• Change in party control, particularly GOP• Growing anti-tax mood.• Policy spillover from K-12 education

WAVE 2 ORIGINS

• Data: 3 states (IN, OH, TN) • Differences from Wave 1 origins

• Greater impact of the economy• Greater role of governors• Greater role of external policy actors e.g.

foundations, Complete College America• Changed motivation of state higher education

boards

Source: Dougherty & Natow (in press)

DEMISE 1: DATA

• Significance: Two-thirds of all states that have established PF later discontinue it at least for a while.

• Data (Dougherty & Natow, in press):• Four states that discontinued PF at some point

and have different PF histories: FL, IL, MO, WA • State with continuing, long-lasting system: TN

DEMISE 2: CAUSES

• Sharp drop in state funding (gross or per FTE). • Call for eliminating PF in order to protect base funding (FL, IL, MO).

• Higher education institution unhappiness: • Hold back system for PF (FL, WA)• Perception that PF used inappropriate indicators (MO, WA)• Perception of insufficient consultation with higher educ (FL, WA)• Perception of erosion of campus autonomy (MO, WA)

• Loss of key government supporters (FL, IL, MO, WA)

• Weakening of business interest in PF (FL, WA).

Source: Dougherty & Natow (in press)

SOURCES

• Burke, J. (2002). Financing Public Colleges and Universities for Performance. Albany: SUNY, Rockefeller Institute.

• Davies, L. (2014). State “shared responsibility” policies for improved outcomes: Lessons learned. Washington, DC: HCM Strategists.

• Dougherty, K. J. & Reddy, V. (2013). Performance Funding for Higher Education: What are the mechanisms? What are the impacts. San Francisco: Jossey-Bass.

• Dougherty, K. J., & Natow, R. S. (In press). The Politics of Performance Funding for Higher Education: Origins, Discontinuations, and Transformations. Baltimore: Johns Hopkins University Press.

• National Conference of State Legislatures. (2014). Performance funding for higher education. Denver, CO: Author. Retrieved from http://www.ncsl.org/issues-research/educ/performance-funding.aspx

FOR MORE INFORMATION:

Please visit us on the web at http://ccrc.tc.columbia.edu,

where you can download presentations, reports,

CCRC Briefs, and sign-up for news announcements.

CCRC is funded in part by: Alfred P. Sloan foundation, Lumina Foundation for Education, The Ford Foundation, National Science Foundation (NSF), Institute of Education Sciences of the U.S. Department of Education

Community College Research CenterInstitute on Education and the Economy, Teachers College, Columbia University

525 West 120th Street, Box 174, New York, NY 10027 E-mail: ccrc@columbia.eduTelephone: 212.678.3091

DENISA GANDARA, UNIVERSITY OF GEORGIAAMY LI, UNIVERSITY OF WASHINGTON

TRAVIS REINDL, BILL AND MELINDA GATES FOUNDATION

POLICY DESIGN

DESIGN PROCESS

• Decision-making locus • Adoption • Development / design

• Stated goal • Fairness / equity for students (e.g., AZ)• Tying funding to state goals (e.g., IL)• Accountability of HEIs (e.g., MN)• Clarity / predictability of funding formula (e.g., AR)

• Instituted mechanisms for revisions to design• E.g., “Refinement Committee” in IL

ACTORS IN POLICY DESIGN IN COLORADO

DESIGN CONSIDERATIONS

• Strength of tool• Base funding / bonus funding / combination• Percentage of funding• Imposed / voluntary • How much weight does budget recommendation carry?

• Sector(s) affected• In TX, currently, only 2-year colleges although 4-year

institutions are developing a performance funding model

• Mission Differentiation • Metrics

• Institutionally-defined metrics• Weights

DESIGN CONSIDERATIONS

• Simplicity (measures and methodology)• 2 metrics in Ohio • 12 metrics for CC in Tennessee

• Transition tools• Phase-In• Hold Harmless• Stop Loss / Stop Gain

• Quality assurance mechanism (learning and labor market outcomes)

• Formula Context• Other funding components (in addition to performance funding)• Other sources of funding for institutions • New funding allocations

• Data collection requirements

DESIGN CONSIDERATIONS

• Metrics and measures employed• Input (public service funding ); process (online course offerings,

decrease energy consumption); outputs (completions, research productivity); outcomes (average wages of graduates)

• Measures • Counts, % improvement, benchmarks (self, institutions in state,

institutions outside state)• Dichotomous v. continuous • Multi-year averages

• Accounting for inputs: input-adjusted outcomes or premiums for underserved populations

• Weights• STEM/critical fields• Underserved populations (e.g., low-income, adult, racial/ethnic

minorities)• Cost of offering program

FREQUENTLY USED METRICS

Intermediate•Credits earned•Success in developmental and gatekeeper courses•Transfer rates•Access and progress for underrepresented students

Ultimate•Degrees awarded (total; per 100 FTE students; underrepresented students•STEM / high demand degrees awarded•Graduation rates•Licensure exam pass rates•Job placement

NICK HILLMAN, UNIVERSITY OF WISCONSIN-MADISONERIK NESS, UNIVERSITY OF GEORGIA

DENISA GANDARA, UNIVERSITY OF GEORGIA

POLICY IMPACTS

WHAT ARE THE IMPACTS OF PERFORMANCE FUNDING?

Study Outcome variable Findings Design

Sample

1 Shin & Milton (2004) Grad rates Null HLM 1997-2001

2 Volkwein & Tandberg (2008) Measuring Up Null Panel 2000-2006

3 Shin (2009) Grad rates & research $ Null HLM 1997-2007

4 Rabovsky (2012) Revenues & expenditures

Mix, mostly null Panel 1998-2009

5 Radford & Rabovsky (2014) Grad & ret rates, degrees

Null, sometimes - Panel 1993-2010

6 Tandberg, Hillman & Barakat (2015) Associate degrees Mix, mostly - or null

Panel 1990-2010

7 Tandberg & Hillman (2014) Bachelor’s degrees Null, some + by year

Panel 1990-2010

8 Hillman, Tandberg, & Gross (2014) Bachelor’s deg (PASSHE)

Null Panel 1990-2010

9 Hillman, Tandberg, & Hicklin-Fryar (in press)

Assoc & cert deg (WA) Null Panel 2002-2012

WHY MIGHT THESE RESULTS OCCUR?

• Studies are of “old” PBF systems, newer is better

• Research design flaws

• Change takes time

• Campuses don’t have will to respond

• Campuses don’t have capacity to respond

• Underlying theory of action is wrong

WHAT IS THE BROADER CAMPUS RESPONSE TO PERFORMANCE

FUNDING? • Dougherty & Reddy (2013)

• Review of performance funding studies on seven state programs

• Primary findings:

• Greater awareness of state priorities

• Increased status competition among campuses

• Increased data use in institutional planning

• Changes in academic and student service policies

HOW ARE CAMPUSES RESPONDING TO TENNESSEE’S OUTCOMES-BASED

FORMULA?• Ness, Deupree, & Gándara (forthcoming)

• Four-campus ethnographic case study of institutional response to Tennessee’s Outcomes-based Funding Formula

• Interviews with 104 campus and system participants

• Observations of 19 events related to campus college completion initiatives

HOW ARE CAMPUSES RESPONDING TO TENNESSEE’S OUTCOMES-BASED

FORMULA?Four main responses:

1.Campus strategic planning

2.Student advising initiatives

3.Administrative streamlining

4.Competition among campuses

WHAT RESEARCH DO WE NEED?

• Student-level analyses

• Alternative outcomes beyond degree completion

• Disentangling the “treatment package”

• Campus implementation studies

• A broader vision/theory of funding reform

ALICIA KINNE-CLAWSON, UNIVERSITY OF WASHINGTONTREY MILLER, RAND CORPORATION

ENSURING AND ENHANCING QUALITY

QUESTION: HOW TO DRAW UPON AND SHAPE THE EMERGING RESEARCH BASE ON COLLEGE QUALITY TO ENSURE THAT

OUTCOMES BASED FUNDING ENHANCES QUALITY?

• Challenge: No consensus on fundamental issues and contentious questions!• How to define quality?• How to measure quality?• How to embed quality metrics in funding models?

• Policymakers: Moving forward and need guidance!

IMPORTANT CONSIDERATIONS FOR DEFINING QUALITY

• Colleges draw on a range of inputs and processes to produce an array of important outputs and outcomes.• Student success; student access and diversity; meeting

workforce needs; research and innovation…

• Colleges have differing missions.• Colleges have different students and resources.• Colleges’ priorities often differ from that of the

state.

CRITERIA FOR QUALITY METRICS

• Flow directly from the needs and goals of students, the state, or the mission and goals of an institution, or a higher-education system.

• Be focused on outputs or outcomes.• Adjust outcomes and outputs for inputs.• Be straightforward and understandable.• Be practical and cost effective to implement.• Provide short- to medium-term feedback.• Be difficult to game.

ADDITIONAL CRITERIA FOR EMBEDDING QUALITY METRICS INTO FUNDING MODELS

• Promote alignment between state priorities and missions and goals of institutions.

• Include and appropriate mix of input, process, output and outcome metrics.

• Account for differences in missions of colleges.

• Ensure that funding tied to metrics is sufficient to gain attention but not so great as to create instability.

• Achieve buy-in of colleges and other stakeholders.

DIRECT MEASURES OF STUDENT LEARNING ARE ATTRACTIVE BUT POSE DIFFICULT CHALLENGES

• Attractive because learning is an important product of colleges that directly addresses critics and can be adjusted for student inputs. Licensure tests are attractive as they measure what graduates need to know to obtain jobs in their fields.

• Challenges include how to balance general vs. specific skills; achieve stakeholder buy-in; and keep costs down?

• Path forward• Short term: use to monitor quality as states implement

outcomes based funding. Licensure tests may be incorporated directly into funding models.

• Long term: conduct research and align stakeholders to possibly move towards larger scale testing.

DIRECT MEASURES OF STUDENT OUTCOMES ARE PRACTICAL AND COST EFFECTIVE, BUT

SHOULD ACCOUNT FOR INPUTS.

• Input-adjusted progression metrics have short- to medium-term time lags, but are easy to game.• persistence towards degree, credit accrual, successful

transfers, credential completion, and acceptance rates at graduate programs.

• Input-adjusted labor outcomes are difficult to game, but have long-term time lags.• measures of job placement, earnings, and student loan

default rates.

• Path forward: Work towards including metrics in funding models.

MEASURES OF STUDENT ENGAGEMENT AND SATISFACTION ARE RELATED TO STUDENT LEARNING AND EXPERIENCES BUT

MORE APPROPRIATE FOR INTERNAL USE BY COLLEGES AND UNIVERSITIES

• Attractive because validated student surveys like the National Survey of Student Engagement (NSSE) capture important aspects about student experiences in college that are related to learning and other outcomes.

• Challenges include implementation costs and achieving buy-in for measures that address process issues that institutions prefer to control.

• Path forward: Encourage institutions to use NSSE and other student surveys to improve the engagement and experience of their students.

STUDENT ACCESS AND DIVERSITY METRICS ARE ATTRACTIVE AND ALREADY INCLUDED IN MANY

OUTCOMES BASED FUNDING MODELS

• Common metrics: Percentages of students receiving or eligible to receive Pell grants, that are ethnic or racial minorities, or that are adult learners.

• Attractive because: Simple, cost effective to implement, provide short-term feedback to colleges and universities, and are difficult to game.

• Path forward: Continue to embed these metrics into outcomes based funding models.

MEASURES OF MEETING WORKFORCE NEEDS ARE ATTRACTIVE BUT REQUIRE FURTHER

RESEARCH

• Attractive because they align institutions’ interests with that of the state; and support a key mission of colleges to improve student labor market outcomes.

• Challenges include a lack of consensus about how to measure progress towards meeting workforce needs; and a lack of research on the merits of different approaches.

• Path forward: Conduct research on how to identify whether institutions are meeting workforce needs, build a consensus around effective approaches, and work towards embedding into funding models.

CONCLUSIONS AND RECOMMENDATIONS

• Work towards embedding into funding models:• Input-adjusted student outcome metrics• student access and diversity metrics. • Licensure tests.

• Consider using student learning metrics and grades to monitor quality and grade inflation during implementation.• Work towards a consensus on learning metrics.

• More research is needed to:• Refine student learning metrics.• Develop good methods to identify workforce needs. • Identify processes that produce desired outcomes and

develop process metrics.

MANY CURRENT OUTCOMES BASED FUNDING MODELS DO ADDRESS QUALITY IMPLICITLY

OR EXPLICITLY

• Many models are designed so as to implicitly include input-adjusted graduates.

• Many models attempt to reward institutions for meeting workforce needs.

• Most models include access and diversity metrics. • A handful of states have successfully implemented

outcomes based funding models that include more explicit quality metrics. • Missouri includes learning metrics; • Nevada and Tennessee include job placement.

FUTURE OF PERFORMANCE FUNDING QUESTIONS FOR REFLECTION

• The Future of Scholarship on Performance Funding • Definitional work

• to enable comparisons and understanding across studies

• Scholarly field agreement on which states have PF programs.

• PF program effectiveness• PF dosage differences• PF formula complexity• PF metrics – are some more effective than others?

• More work on why PF programs end.• Remaining frontier - states that have never adopted a PF

policy.

FUTURE OF PERFORMANCE FUNDING QUESTIONS FOR REFLECTION

• The Future of Performance Funding as a Policy• It is difficult to be against PF as a policy politically, but…

• Will variance in PF implementation continue to make the policy hard to define ?

• Is there a PF 3.0?• If PF remains mostly marginal funding, what expectations are

there for change/effectiveness/longevity?• What elements would you include in this PF design?

Jennifer DelaneyUniversity of Illinois at Urbana-Champaign