Upload
wade-mcintosh
View
43
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Environmental Policy Evaluation. MADE Program 30/10/2009. Policy Evaluation. Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary) - PowerPoint PPT Presentation
Citation preview
Environmental Policy Evaluation
MADE Program
30/10/2009
Policy Evaluation
• Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary)
• Voluntary programs/International Agreements (ISO 9001/Kyoto Protocol) are greatly increasing in number and size
• Need for rigorous, scientfic evaluations of policies
Treatment
• Policy is known as the treatment
• Observations of unaffected entities is called the control
• An effective policy will alter the outcome of those treated (given the policy) relative to the control
Policy Story
• It is worthwhile to learn the policy background before starting an evaluation
• Sometimes the proper outcome variable to measure is not clear
• Be willing to listen to those involved in the policy but remember their interests are different than yours – No Stockholm Syndrome
Ideal Story
• You find data that:– Starts before policy went into affect– Continues after policy went into affect– Covers those affected by policy and those
unaffected
• The policy– Randomly assigned those involved– Staggered its starting date
Issue to Control for
• The big issues to control for are:– Pre-policy behavior/trend– Behavior of those unaffected by policy– Selection of those affected by policy– Other variables that affect outcome
• Ideally the policy will have a staggered start so that the analysis is using both time and space variation to determine policies affect
Panel Data
• Usually evaluations occur using panel data methods
• Panel data is when you follow the same entity over time
• Methods often used are fixed effects, random effects, dynamic panel, logit
Voluntary Programs
• Generally between government agency and a group of firms
• Firms join the program (become a partner) and pledge to make specified environmental improvement
• Partners potentially receive benefit in form of confidential information
Standard Analysis
• Conventional interpretation of partnership programs is that information given to partners bring exclusive benefits
• Selection bias likely to occur; needs to be controlled for
• Typical View - Program effective if partners had statistically better environmental outcome than non-partners
Estimation• Difference in Difference Model
– Post Policy Dummies (T)– Partner Dummy (P)– Partner & Policy Dummy Interaction (TP)– Other controls that alter dependent variable
(X, S, N)
• β9 is the relevant variable
itiititititjtitit vPTTPNSXR ^
987654
Diff-in-Diff Variable
• β9 tells us whether the group affected by the policy acted differently than the group unaffected, once the policy went into affect
• If it is positive & statistically significant, the policy is associated with an increase in dependent variable
• Vice Versa for negative coefficients
Other Interpretations
• Policy variable (β8) tells us whether the dependent variable changed when policy went into affect
• Partner variable (β7) tells us whether partners are inherently different with respect to dependent variable
13
Example: C2P2 program• Coal Combustion Partnership Program (C2P2) started
in 2001, part of Resource Conservation Challenge– promotes beneficial uses of Coal Combustion Products (CCP)
as opposed to lanfilling them– CCP can be reused in concrete, asphalt, & more
• About 170 Partners– Work with industry groups & other federal agencies to promote
education about CCP reuse– Outreach/networking/acknowledgement for CCP suppliers and
demanders – Document reuse by individual facilities, companies and
government organizations
Question• Is C2P2 associated with an increase in CCP
re-use?
• Data on coal power plants in US for 96-05 (C2P2 started in 2001)
• Dependent Variable: Fly Ash Re-Use Rates – the amount of fly-ash sold divided by the total
fly-ash generated
15
Rate of Fly Ash Reuse
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Re-
Use
Rat
io
Initial Partners
Non-Partners
Late Partners
Raw Data
• Be careful with Raw Data like this
• Other co-variates not controlled for
• Policy makers tend to like these pictures even though they are not telling the whole story
17
Estimation• Random Effects Model
• Dependent Variable:– Ratio of Fly Ash Reused to Total Fly Ash Generated
• Controls:– Pre, Early, & Late C2P2 Time Dummies – Partner Variable & C2P2 Time Interactions– Total Coal Burnt– Ash Content of Coal– Price of Aggregates/Cement in State– Level of Cement Imports– Region Dummies– NOx Control Equipment Present
itiititititjtitit vPTTPNSXR ^
987654
18
Evaluation ResultsTime Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorPredicted Partner -0.18** 0,08Early-C2P2 (2001-2002) 0.06** 0,02 0.08** 0,03Late-C2P2 (2003-2005) 0.06** 0,02 0.07** 0,02Early-C2P2* Predicted Partner 0,03 0,02 0,03 0,02Late C2P2* Predicted Partner -0.01 0,02 -0.01 0,02Number of Utility's Plants Near 0.01* 0 0,01 0,01Post-C2P2* Utility's Plants Near 0,01 0,01 0,01 0,01Lag Utility Reuse 0.08** 0,04 0.11*** 0,03Aggregates Price -0.01 0,02 0.06*** 0,02Cement Price 0,01 0,01 -0,01 0,01Cement Imports 0,01 0,01 0,01 1SCR Installed 0,01 0,02 0,01 0,01Avearge Ash Content -0.01*** 0 -0.02*** 0,01Total Coal 0.01** 0,00 0.01*** 0,00N 1883 1883Plants 303 303R2 0,04 0,3Region & Year Dummies Not Shown for BrevityErrors Clustered by Utility for all specifications and Bootstrapped for the first two specifications*, **, *** indicate 10%, 5% and 1% statistical significance, respectively
Fly Ash Reuse RatioFly Ash Reuse Ratio
Other Issues
• Treatment Spillovers
• Selection Bias
• Error Specification
20
Treatment Spillovers• Programs provide information that spills-over
to non-partners; programs may be public good in nature
– Technical information-spreading is not limited to partners, thus both partners and non-partners can take advantage of program
– One country agreeing to reduce consumption of a good could lead to increases in price for other countries
Evaluation Interpretation
• If there are treatment spillovers,– Partners and non-partners do not need to be
statistically different for program to be effective– Requires evidence that spillovers exist– Requires statistically better environmental
outcomes for both groups after program initiation
22
Evidence of Spillovers: C2P2
• Non-partner data analyzed for evidence of treatment spillover by comparing non-partners in states w/ many partners to non-partners in states w/ few partners
Example: C2P2Time Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorModerate/Low C2P2 Partners Nearby 0.34*** 0,08Early-C2P2 (2001-2002) 0.10** 0,04 0.11** 0,04Late-C2P2 (2003-2005) 0.14** 0,05 0.15** 0,05Early-C2P2* Moderate/Low C2P2 Partner -0.09* 0,05 -0.09* 0,05Late-C2P2* Moderate/Low C2P2 Partner -0.14** 0,06 -0.14** 0,06Number of Utility's Plants Near -0.01 0,01 -0.01 0,01Post-C2P2* Utility's Plants Near 0,01 0,02 0,01 0,02Lag Utility Reuse 0,07 0,06 0,04 0,06Aggregates Price -0.01 0,02 -0.01 0,02Cement Price -0.01 0,01 -0.01* 0,00Cement Imports 0,01 0,01 0,01 0,01SCR Installed 0,04 0,03 0,04 0,04Avearge Ash Content -0.01** 0 -0.02** 0,01Total Coal -0.01 0,01 0.01** 0,00N 1088 1088Plants 176 176R2 0,2 0,48State & Year Dummies Not Shown for BrevityErrors Clustered by Utility*, **, *** indicate 10%, 5% and 1% statistical significance, respectively
Fly Ash Reuse Ratio Fly Ash Reuse Ratio
Selection Bias
• If entities are not randomly assigned to treatment or control, selection bias could be a problem
• If treatment is mandatory, selection bias is less likely to be a problem
• If treatment is voluntary, then selection bias is likely to be a problem
Endogeneity
• Another way to describe selection bias is to say that the choice of enetering the voluntary progam is endogenous
• This generally means that the choice of entering the program affects the outcome measure, rather than the outcome measure affecting the choice of entering the program.
Direction of Bias
• One big problem is that the direction of bias could be either positive or negative
• If entities join program because it is easy to acheive goals set, the bias is positive
• If entities join program because they really need the help, the bias is negative.
Example-Positive
• Norway joins the Oslo Protocol because it has large natural gas reserve, making reducing sulfur emissions relatively cheap
• If evaluate the effect of the Protocol on Norway (relative to Poland for example), it will look like the Protocol had a large effect when in reality the change in behavior is driven by the switch to natural gas.
Example-Negative• Poland joins the Sophia Protocol even
though it relies heavily on coal for electricity production
• If we evaluate the effect of the Protocol on Poland (relative to Sweden for example) it will look like the Protocol is having no effect when in reality it is the lack of alternatives for electricity that is driving the result.
Correcting for Endogeneity
• Generally known at Instrumental Variables
• Find other variables that predict the variable that is endogenous
• In the case of selection bias into the policy treatment, this generally means variables that are associated with the benefits or costs of recieveing the treatment
30
Partner Choice Example: C2P2• Dependent Variable:
– Partner Designation
• Controls:– Investment in Solid Waste Disposal 96-00– State w/ CCP re-use authorized– State w/o CCP Permit Requirement– Utility Size– Avg Fly and Bottom Ash Reuse 96-00– Avg Price of Agg. & Cement in State 96-00
31
Example : C2P2Dependent Variable: Model: ProbitExplanatory Variable Coefficient Stnd. ErrorAvg Fly Ash Reuse Ratio 1996-2000 -0.11 0.28Avg Bottom Ash Reuse Ratio 1996-2000 0.2 0.24Avg Aggregates Price 1996-2000 0.34*** 0.08Avg Cement Price 1996-2000 -0.01 0.01Utility Size 0.04*** 0.01Restructed Market -0.41* 0.22NOx SIP Call State 0.15 0.5Total Solid Waste Disposal Investment 1996-2000
0.03 0.02
No CCP Permit Required 0.45** 0.21Re-use Guidelines Set 1.1*** 0.3N 305R2 0.18*, **, *** indicate 10%, 5% and 1% statistical significance
Initial Partner Dummy
Error Specification
• Generally in panel methods, you would like to cluster errors by largest grouping entity possible
• These could be industry classification, firm (as opposed to plant), country, etc
• This controls for unobserved correlation better than a narrow grouping
Other Options
• Bootstrapping of error terms is a catch all – Code is quite easy in Stata to code
• Many other potential fixes available, Stata help very good for this
Policy Evaluation
• Important to:– Consider pre-policy trends– Consider control group’s actions– Other co-variates– How entities enter the policy– How the treatment affects entities
• With these issues covered, you have done as much as is possible as a data user
Questions?
• About Data
• About Methods
• About Examples
• About Information on Policies
• About dealing with policymakers
• …