Embed Size (px)
Citation preview
ECONOMIC TOOLS TO EVALUATE SOCIAL SCIENCE
PROGRAMS
BIDISHA MANDALSCHOOL OF ECONOMIC SCIENCESWASHINGTON STATE UNIVERSITY
WSAC, 2011 Extension Directors Conference
November 16, 2011
Overview
Economic tools Why evaluate? What are economic tools, economic analyses? When to evaluate? How to evaluate?
Case studies from Health Extension Spokane Public schools: Nutrition services
intervention for middle school students Strengthening Families Program: Substance abuse
prevention for adolescents Providence health care services: Transitional care
model to reduce preventable hospital readmissions Supplemental Nutrition Assistance Program
Education: University of Idaho Extension
Magnitude of the Problem
Primary prevention Health promotion
Secondary prevention Screening, diagnosis, therapies
Tertiary prevention Treatment to prevent or postpone
complications
A Crucial Difference
Program effectiveness Outcome oriented Directly links the intervention with health
outcomes of interest
Program evaluation Ways to maximize the intended impact with
available resources, or Ways to obtain a particular impact with as
little resources as possible
Importance of Evaluation
Resource constraints To eliminate or reduce waste Evidence of return on investment
Ideally… Inform decisions Help make choices about future allocations
Example: Health-care system Getting value for money is a widely accepted and
legitimate goal Quality medical care in part translates into
potentially expensive demands for new drugs and technology
Pressure to improve efficiency, make trade-offs, and develop incentive systems for patients and physicians while holding down healthcare costs
Address Two Questions
What works? Multiple interventions could work
What works best? Identify the one intervention that provides the
greatest bang for the buck! Example: Increased prevalence of diabetes
among adults Strategies and reasonable alternatives
Physical activity – Fitness programs (worksite, community, less TV/computer time)
Diet – cooking programs, education program to change food consumption behavior
What is the objective Increased physical activity? Short-term Reduced risk of diabetes? Long-term
Economic Analysis
Evaluation is part of the program design and planning There are always competing use of resources Frame the study in order to consider
opportunity costs for each of our choices Identify, measure, value and compare the
costs and consequences of alternative prevention strategies
Quantitative and analytic methods Cost-benefit analysis Cost-effectiveness analysis Cost-utility analysis
Identify the Risks Factors
Define the target population for the intervention
Define the problem or question, and magnitude of impact
Define the information needs of the target population in reference to the program or intervention
These steps will Influence the types of benefits and costs to be
included Help to determine which analytic method is
most appropriate
Identify the Intervention(s)
Indicate clearly the preventive strategies under consideration, including baseline if any
Specify perspective of the program and analysis
Limit perspectives to those relevant to the study
Define relevant time frame in which program will be delivered
Determine how far into the future costs and effects that accrue from the intervention will be considered
Background on the Intervention Can it work?
Efficacy Degree to which intervention strategies can work under ideal
conditions, with carefully selected participants, and optimal resources. Example: Randomized controlled trials
Will it work? Effectiveness How well these strategies actually work in community
settings Demonstrates real-world effectiveness under practical
resource constraints Effectiveness is likely to be lower than efficacy
What are the benefits and costs of the intervention? Units of measurement How do benefits compare with costs? What additional benefits could be obtained with additional
resources?
Identify the Methods
Determine the analytic methods for decision-making
The choice will depend on the policy question, the outcomes of interest, and the availability of data
Determine whether analysis is to be marginal or incremental Marginal analysis: Examines the effect of
expanding or contracting an intervention Incremental analysis: Compares the effects of
alternative programs
Identify the Outcomes
Identify the relevant costs Program costs Productivity losses
Identify the relevant outcomes Number and nature of health outcomes
Specify the discount rate or time preference for monetary and non-monetary costs
Identify sources of uncertainty and plan sensitivity analysis
The Role of Discount Rate
Individuals generally weight costs and benefits in the near future more heavily than in the distant future
This applies to the valuation of capital and investments and to health outcomes
Societal preference is for health benefits received today versus health benefits received in the future
Using an appropriate discount rate in an economic analysis allows Adjusting the value of receiving benefits today versus in
the future or of incurring costs today versus in the future Makes benefits and costs comparable over time
Discount rate is selected based on the study perspective Social or private or individual
The Role of Uncertainty
Precise estimates of costs and benefits/effects are often not available Limited literature Different population settings
Important to list all assumptions upon which estimation is dependent
Perform sensitivity analyses How will result of evaluation change if the
assumptions change?
Evaluation Tools
3 most commonly used techniques Cost-benefit analysis (CBA) Cost-effectiveness analysis (CEA) Cost-utility analysis (CUA)
Each method Allows comparison of different intervention
strategies Calculates resources consumed and outputs
generated Requires quite similar cost analysis Assessment of outcomes, both benefits and harms
(negative benefits) Scope of analysis determines the appropriate analytic
method
Cost-Benefit Analysis
All costs and benefits valued in dollars Costs include
Cost of program Cost to participants - out-of-pocket expenses,
productivity losses, travel time, child care, intangible costs (pain, suffering)
Benefits include All types of beneficial and harmful health
outcomes, whether intended or not Have to be valued in monetary terms
CBA is well suited to comparisons with interventions that include cross-sector considerations Housing, education, transportation
interventions
Cost-Effectiveness Analysis
Usually examines direct medical, non-medical, and productivity costs
Compares costs with outcomes in standard health units Example: costs per case averted
Most suitable when comparing interventions that have similar health outcomes
Cost-Utility Analysis
Modified version of CEA Compares direct medical and non-medical
costs with health outcomes converted to a standard health unit, often a quality adjusted life year (QALY) combining both mortality and morbidity
Often used to compare health intervention which have different type of health outcomes
CASE STUDIES
Spokane Public Schools:Objective Spokane public schools’ lunch program
With Doug Wordell, Ruth Bindler, Kenn Daratha, Sue Butkus Intervention program included reducing vending machine
beverages, limiting ala carte offerings, and adding seasonal fruits and vegetables to student lunch menus
Compare pre-program and post-program behavior
Objective If there were associations between an altered school food environment and
food choices of middle school students both in and outside of school My involvement
Retroactive, after program was delivered and surveys were conducted
My role Analyze survey data Determine food behavior change
Spokane Public Schools:Method & Results Results
Healthful modifications in the school food environment associated with some positive food behaviors
The cost of conducting the intervention was approximately $24,000/year Lost ala carte sales, loss in vending machine
sales More expenditure on produce
In this study, is it possible to show the benefits of improved food choices outweigh the costs? NO - Related data was not collected for CBA
Spokane Public Schools:What did I learn? Difference between outcomes and
impacts In this study, there is no way to link
improvement in behavior to improvement in health (short-term or long-term)
Even if they are positively related, we have no quantitative measure for the benefits
Economic analysis is not always possible Unless evaluation is part of the program
design
Strengthening Families Program:Objective Strengthening Families Program (SFP) for Parents
and Youth 10-14 years With Laura Hill, Robby Rosenman, Ron Mittelhammer Voluntary, family-based intervention Designed to discourage future substance abuse among
adolescents and youth Compare pre-program and post-program behavior
Randomized clinical trials (RCT) have shown SFP to be cost-effective and that benefits outweigh costs
How does SFP’s impact in community dissemination compare to results from RCTs?
My involvement Retroactive, after programs were delivered and surveys
were conducted
Strengthening Families Program:Method & Results Community dissemination has many practical issues
Variation in program delivery across counties, states Data not recorded systematically or consistently
CBA, CEA and CUA not possible Do not have necessary data
But, have data to determine which factors in community dissemination of the program are different Attrition – high incompletion rates, some people come to the
sessions but do not respond to surveys Self-selection – more motivated parents are more likely to
attend SFP Results
People who come to the sessions but do not respond to surveys have lower self-assessment scores
Strengthening Families Program:What did I learn? Validity of assumptions
Does sample match population? Differential dropouts Are facilitators similar? Some sessions have additional orientation
session Language of delivery
Providence Hospital Transitional Care:Objective Transitional care model
With Cindy Corbett Innovative model to improve and synchronize hospital
discharge planning and deliver core transitional care intervention to patients at high risk for potentially preventable readmissions
Secondary objective: document barriers and facilitators of successful delivery in different hospital environments
My involvement Contacted during the planning process But, resource constrained
Unable to collect/record some necessary data for a complete CEA Future studies will look at CUA
Providence Hospital Transitional Care:Method & Results Cost-effective analysis of transitional care model
Compared to patients not receiving transitional care Have necessary costs
Cost of transitional care nurse, inpatient pharmacy, pharmacotherapy clinic, administrative costs, home health care costs, hospital care costs
Have some necessary effects (in $) Hospital care avoided due to lower re-hospitalizations Revenues from inpatient pharmacy, pharmacotherapy clinic
not recorded Result
Decrease in re-hospitalizations and ED visits Total savings over 4 months = $55,752.34 (for ~ 100
patients) Savings likely to increase over time since some of the cost
items were fixed costs, and some revenues were unknown
Providence Hospital Transitional Care:What did I learn? Know your audience
Who is using the results? What is the study perspective
Results of this study to be used by hospital administrators CBA not appropriate
Supplemental Nutrition Assistance Program:Objective University of Idaho Extension, Nutrition and Food Safety
With Shelly Johnson, Joey Peutz and others Follow Virginia Tech report for CBA (1996) to calculate the
costs and benefits of UI’s Supplemental Nutrition Assistance Program Education (delivered in Coeur d’ALene)
My role A complete CBA with sensitivity analysis
My involvement: Quite proactive Involved in program design – control and intervention Introduce new questions to pre- and post-program surveys to
improve CBA
Supplemental Nutrition Assistance Program:Method & Results Costs and benefits
Collect up-to-date information on costs of health conditions/diseases averted due to improvement in nutritional intake and health behavior
Cost of program delivery (compared to control group)
Results Control and intervention will take place in
February/March 2012 Analysis results expected middle of next year
Supplemental Nutrition Assistance Program:What did I learn? Practical issues
Participant enrollment Differences in incentives to participants in
control and intervention groups Sample size for robust results Are cost data in literature suitable for Idaho
population?
Challenges
Time consuming process Uncertain monetary values in CBA Uncertain QALY values in CUA Comparison of results under different
situations Validity of assumptions
Reference
Haddix, A.C., Teutsch, S.M., Corso, P.S. (2003). “Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation”. Second Edition, Oxford University Press.
