1

The Role of Performance Management in Workforce

Investment Programs

Burt S. BarnowInstitute for Policy StudiesJohns Hopkins University

Prepared for the Conference What Europeans Can Learn from the WIA Experience

Washington, DC November 7, 2009

Topics Covered Background on performance management Elements of a performance management

system Performance Management and Evaluation Performance management in U.S. workforce

programs Adjustments to performance standards Evidence on the effects of performance

standards Conclusions and lessons

2

Background Why have performance management?

Principal-agent problem: interests of state and local governments may differ from those of federal government

Local government may have own goals, e.g., maximize participants, appeal to maximum voters, provide work to local employees

JTPA and WIA systems tried to align goals of federal, state, and local government

Problem: goals of program not easy to identify: equity vs. efficiency for federal government

3

Elements of a Performance Management System Performance Measure(s): The indicator of success used to judge

performance. Measures can be: Inputs (number of people served, hours of instruction) Process variables (characteristics of a course, procedures followed) Gross outcomes ((program completers, placements, no. passing a test) net outcomes (skills added by the program) Value of outcomes (earnings gains)

Performance Standards and a means to set the standards Standards can be absolute or relative Standards can be set objectively or subjectively Standards can be the same for all or they can be adjusted

Rewards and Sanctions are needed to encourage meeting or exceeding the standards but not always emphasized

4

Distinctions between Performance Measurement and Evaluation

Performance measurement is a management tool to track progress over time

Evaluation answers questions about whether program is meeting expectations

What happened? (Process Study) What difference did it make (Impact Study) Was it worth it? (Cost-benefit Study)

Evaluations can be used in management, but on a slower path than performance measurement

We need both types of activities, and it is important not to confuse them

5

The JTPA Performance Management System The Secretary of Labor issued the performance

measures and set national standards as well as an adjustment procedure

The national standards were set so that based on past experience, 75 percent of the SDAs were expected to meet or exceed the standard.

The adjustments were determined from regression models estimated on management information system data submitted annually by each SDA. Variables with insignificant coefficients or coefficients with an unexpected sign are omitted from the adjustment model.

The performance measures were gross outcomes

6

JTPA Measures and National Standards

The adult follow-up employment rate, defined as the proportion of adult respondents who were employed at least 20 hours per week during the 13th week after termination (59 percent);

Adult follow-up weekly earnings, defined as average weekly earnings for all adults who were employed for at least 20 hours per week during the 13th week after termination ($281);

Welfare adult follow-up employment rate, defined in the same manner as for the adult follow-up employment rate but for adult welfare recipients only (50 percent);

Welfare follow-up weekly earnings, defined in the same manner as adult follow-up weekly earnings ($244);

Youth entered employment rate defined as the proportion of youth terminees (other than potential dropouts who remained in school) who entered employment with at least 20 hours per week (41 percent); and

Youth employability enhancement rate, defined as the proportion of youth who obtained one of the employability enhancements at termination (40 percent).

7

Performance Standards under the Workforce Investment Act Standards now set for states as well as local areas Standards now “negotiated” at state level rather than based on model States determine standards for local areas WIA now uses “Common Measures” established by OMB for workforce-

oriented programs; for adults these are: Entered employment rate: Of those not employed at the date of

participation, the number of participants who are employed in the first quarter after the exit quarter divided by the number of participants who exited during the quarter.

Employment retention rate: Of those who are employed in the first quarter after the exit quarter, the\ number of participants who are employed in both the second and third quarters after the exit quarter divided by the number of participants who exited during the quarter.

Average Earnings: Of those participants who are employed in the first, second, and third quarters after the exit quarter, total earnings in the second quarter plus total earnings in the third quarter divided by the number of participants who exited during the quarter.

8

Why Adjust Performance Standards?

Better measure value added by programs rather than gross change

Level the playing field (horizontal and vertical equity)

Discourage (but not eliminate) gaming responses for programs with discretion in who is served and how

Provide more accurate and useful information program performance management and accountability

9

Reasons not to AdjustPerformance Standards

Set more ambitious standards for low achievers Promote single standard of minimum performance

Ex.: Set same standard for reading and math proficiency for disadvantaged children as we do for relatively more advantaged?

Regression-based adjustments more difficult for public officials to understand

Explanatory power of adjustment models often low, limiting confidence in results

Rigid application of statistical adjustment models may contribute to unintended consequences, distortions

10

Evidence on the Effects of Performance Standards JTPA system led to modest cream skimming; WIA has stronger

incentives for cream skimming due to absence of regression adjustment model

Short-term performance measures on earnings levels have little relationship to long-term impact measures

Limited empirical evidence on impact of performance standards on services provided, but believed that local areas try to maximize measured performance; one study finds that effects on efficiency are mixed

No evidence identified on effects of performance management on technical efficiency of service delivery

Performance standards lead to real resources being spent on strategic behavior by local programs.

11

Conclusions and Lessons Do not confuse performance measurement with program evaluation There are often good reasons to adjust performance standards to take

account of program goals, participant characteristics, and environmental conditions

Programs need not have the same performance measures or standards

Be cautious in establishing performance measures with large rewards and/or sanctions

Efficiency measures are particularly prone to programs avoiding more disadvantaged customers and more costly services

Performance management Is still in a formative stage; Legislation should not be overly prescriptive on the measures,

standards, and incentives. Deliberations on the structure of the performance management

system should include input by all the relevant stakeholders

12

The Origins of Performance Measurement?

“All lovers swear more performance than they are able, and yet reserve an ability that they never perform; vowing more than the perfection of ten, and discharging less than the tenth part of one.”

--William Shakespeare, Troilus and Cressida

13