Upload
faolan
View
38
Download
3
Tags:
Embed Size (px)
DESCRIPTION
The Intersection of Performance Measurement and Program Evaluation: Searching for the Counterfactual Moscow 2011 . Douglas J. Besharov School of Public Policy University of Maryland. Performance Management. Efficiency studies (“outputs”) How much does the program cost? - PowerPoint PPT Presentation
Citation preview
The Intersection of Performance Measurement and
Program Evaluation:Searching for the Counterfactual
Moscow 2011
Douglas J. BesharovSchool of Public PolicyUniversity of Maryland
Performance Management
4 Efficiency studies (“outputs”)4 How much does the program cost?
4 Monetary, nonmonetary, and opportunity costs4 Could it be delivered more efficiently?
4 Effectiveness studies (“outcomes” and “impacts”)4 Does the program achieve its goals?4 Could it be more effective?
4 Both require a comparison, or a “counterfactual”
8Douglas J. Besharov, June 2011
Appearances Can Be DeceivingGiving children a “Head Start”
Douglas J. Besharov, June 2011 2
It Matters How Children Are Raised
Douglas J. Besharov, June 2011 3
Ineffective Early Childhood Education ProgramsIHDP
1985-1988
• Low-birth weight, pre-term infants and their parents
• Home visits, parenting education, and early childhood education
• $20,400 per child per year
• No significant impacts, initial IQ gains fade
CCDP1990-1995
• Poor children under age 1 and their parents
• Case management, parenting education, early childhood education, and referrals to community-based services
• $19,000 per family per year ($60 million annually)
• No significant impacts
Early Head Start1996-2008
• Poor children ages 0-2 and their parents
• Child development, parenting education, child care, and family support services
• $18,500 per child per year ($700 million annually)
• No significant impacts
Douglas J. Besharov, June 2011 4
Douglas J. Besharov (October 21, 2008)
“The closest thing to immortality on this Earth is a federal government program.”
– Ronald Reagan
Program Improvement, not Program Dismantling
Performance Management
4 Leadership,
4Management, and
4Measurement
6Douglas J. Besharov, June 2011
Performance Management
4 Leadership,
4Management, and
4Measurement
7Douglas J. Besharov, June 2011
Point #1
Counterfactuals are needed for accurate performance measurement.
9Douglas J. Besharov, June 2011
4 Head Start Impact Study (2010): 10 years and running
4 Moving to Opportunity Study (1994): 17 years and running
4 Employment Retention and Advancement evaluation (1998): 13 years and running
4 Building Strong Families Project (2002): 9 years and running
4 National Job Corps Study (1993): 15 years to complete
Impact Evaluations Take Too Long
to Manage Performance
Douglas J. Besharov, June 2011 10
Logic Model for Job Training Programs
Inputs
Training facilities
Staff
Funding
Client characteristics
Activities
Job search/job readiness training
Classroom instruction
Job skills training
Outputs
Hours of training instruction
Hours of practice
Staff admin
Skill certificates
Outcomes
Job search skills
Technical job skills
Interpersonal skills
ProximalImpacts
Earnings Employment
UI/Welfare Receipt
Crime
DistalImpacts
Higher lifetime
earnings/employment
Lower poverty
Stronger economy
Theory: If government provides job training to the unemployed, than the unemployed will receive job skills necessary for good jobs, increased earnings, and a stronger economy
Design: (1) Job search/job readiness training, (2) skills training, (3) in a classroom.
External Community and Societal Context
Problem: Some unemployed do not have the necessary skills to obtain and keep well-paying employment, leading to lower income, greater use of government benefits, and a weaker economy.
Point #2
Carefully applied, a measured outcome coupled with a logic model’s theory of change
—often buttressed by other evidence—can serve as a more timely and more
useful performance measure than a formal evaluation of long-term impacts.
12Douglas J. Besharov, June 2011
4 There is no output, so no positive outcome can be reasonably predicted.
4 The output itself is sufficiently suggestive of a likely outcome.
4 The output is produced at such a prohibitively high cost, that, regardless of its likely outcome, it does not meet cost-effectiveness or cost-benefit tests.
When “Outputs” Imply “Outcomes”
Douglas J. Besharov, June 2011 13
4 Evaluations of on-going programs4 Rolling randomized experiments
4 Pre-post studies (with embedded counterfactual)
4 Regression-discontinuity designs
4 Evaluations of specific program “improvements”4 Randomized experiments
4 Pipeline studies (or rolling implementation)
4 Interrupted time series studies
Feasible “Outcome” Evaluations
Douglas J. Besharov, June 2011 14
A Clear Interrupted Time Series
Douglas J. Besharov, June 2011 15
Circling the Wagons
Douglas J. Besharov, June 2011 16
4 Top-down administrative and funding incentives
-- together with --
4 Bottom-up voucher-like programs
Accountability Systems
Douglas J. Besharov, June 2011 17
Appearances Can Be DeceivingGiving children a “Head Start”
Douglas J. Besharov, June 2011 2
When “Outcomes” Imply “Impacts”
4When the desired impact is reasonably predicted to follow from the measured outcome
Douglas J. Besharov, June 2011 10
Teen Pregnancy in Anson County, NCTeen Pregnancy in Anson County, NC20012001--20082008Scale: 0Scale: 0--100100
0
20
40
60
80
100
2001 2002 2003 2004 2005 2006 2007 20080
20
40
60
80
100Rate per 1,000
Adolescent Parenting Program
Number of pregnancies
Ineffective Job Training ProgramsJob Corps
• Low-income youth
•
•
JTPA1987-1994
• Low-income adults, dislocated workers, and out-of school youth
• Classroom training, on-the-job training, job search assistance, adult basic education, and other services
• $2,400 per participant for 3-4 months ($60 million annually)
• Women: Small initial gains in earnings, employment, and GED receipt fade by 5 yrs
• Men: Small initial gains in earnings fade by 5 yrs, no other impacts
• Youth: On significant impacts
WIA (dislocated)2003-2005
• cts
Douglas J. Besharov, June 2011 2