A Theory of Evidence for EvidenceBased Policy

A Theory of Evidence for Evidence-Based Policy

Page 1 of 30

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

UniversityPressScholarshipOnlineBritishAcademyScholarshipOnline

Evidence,InferenceandEnquiryPhilipDawid,WilliamTwining,andMimiVasilaki

Printpublicationdate:2011PrintISBN-13:9780197264843PublishedtoBritishAcademyScholarshipOnline:January2013DOI:10.5871/bacad/9780197264843.001.0001

ATheoryofEvidenceforEvidence-BasedPolicyNANCYCARTWRIGHTJACOBSTEGENGA

DOI:10.5871/bacad/9780197264843.003.0011

AbstractandKeywords

Evidence-basedpolicyisalltheragenow.Butnooneknowsquitehowtodoit.Policyquestionsdonotgenerallyfallneatlywithinanyoneofourscientificorsocialsciencedisciplines,wherethestandardsandrulesofevidenceforthequestionsstudiedarefairlyclearlydelineated.Thereisbynowavarietyofguidesavailableonstandardsofevidenceforevidence-basedpolicy.Butthesefocusnarrowlyononlypartoftheproblem.ThischapterlaysthefoundationsforaguidefortheuseofevidenceinpredictingpolicyeffectivenessinsituamorecomprehensiveguidethancurrentstandardofferingssuchastheMarylandrulesincriminology,theweightofevidenceschemeoftheInternationalAgencyforResearchonCancer,ortheUSWhatWorksClearinghouse.Theguideitselfismeanttobewell-groundedbutatthesametimetogivepracticableadvice,thatis,advicethatcanbeusedbypolicy-makersnotexpertsinthenaturalandsocialsciences,assumingtheyarewell-intentionedandhaveareasonable


Page 2 of 30


butlimitedamountoftimeandresourcesavailableforsearchingoutevidenceanddeliberating.

Keywords:evidencetheory,policyeffectiveness,evidenced-basedpolicy

AbstractEvidence-basedpolicyisalltheragenow.Butnooneknowsquitehowtodoit.Policyquestionsdonotgenerallyfallneatlywithinanyoneofourscientificorsocialsciencedisciplines,wherethestandardsandrulesofevidenceforthequestionsstudiedarefairlyclearlydelineated.Thereisbynowavarietyofguidesavailableonstandardsofevidenceforevidence-basedpolicy.Butthesefocusnarrowlyononlypartoftheproblem.Forpolicywewantcredibleevidencethatspeaksfor(oragainst)thepolicyandwewanttoknowhowtoevaluatepolicyeffectivenessinlightoftheevidence.Thissuggeststhreequestions:(1)Whenareevidenceclaimscredible?(2)Whenisacredibleclaimrelevanttothetruthofaclaimtoeffectiveness?(3)Whatistheprobabilitythatapolicywillbeeffectivegivenabodyofevidenceofvaryingcredibilityrelevantindifferentways?

Currentguidestendtofocusonquestion1,rankingevidenceclaimsaccordingtothemethodsbywhichtheclaimsareproduced,thatis,accordingtohowmuchcertaintythemethodconfersontheclaim.Inanattempttoaddressallthreequestionsinonefellswoop,thispaperstartsnotfromquestion1,butfromquestion3.Asure-firewaytoevaluatewhattheeffectsofapolicyimplementationwillbeistomimictheproceduresbywhichnatureherselfdecideswhateffectstoproduce:surveyallthecausesforthetargetedeffectthatareinplaceinthetargetsituation,includingthoseyouintroduceinimplementingthepolicy;thenpredicttheeffectfromnaturesownrulesforcalculatingwhathappenswhenthosecausesactinconsort.Answerstoquestions1and2canthenbegearedtothisstrategy.Ofcoursemimickingnatureproperlyisoutofthequestion.Butwesuggestfirstthatdoingsowellenoughmaynotbesuchatallorderandsecondthatwhetherwewanttomimicnaturesmethodornot,whenwebetthatapolicywillbeeffectivewearewilly-nillybettingonsuchasimulation.

(p.292) I.Preliminaries

TheprojectWEAIMHEREtooutlineatheoryofevidenceforuse.Morespecificallywelayfoundationsforaguidefortheuseofevidenceinpredictingpolicyeffectivenessinsitu,amorecomprehensiveguidethancurrentstandardofferings,suchastheMarylandrulesincriminology,theweightofevidenceschemeoftheInternationalAgencyforResearchonCancer(IARC),ortheUSWhatWorksClearinghouse.Theguideitselfismeanttobewell-groundedbutatthesametimetogivepracticableadvice,thatis,advicethatcanbeusedbypolicy-makersnotexpertinthenaturalandsocialsciences,assumingtheyarewell-intentionedandhaveareasonablebutlimitedamountoftimeandresourcesavailableforsearchingoutevidenceanddeliberating.


Page 3 of 30


Wegointotheprojectwithsomeassumptions.Thefirstisadelimitationofthetopic.Theguideforwhichweaimtolayatheoreticalbaseistobeconcernedwiththeuseofevidencetoestimate,ifonlyroughly,whether,wereaproposedpolicytobeactuallyimplemented,aspecific,identifiedoutcomewouldbeproduced.Wethusdonotdiscussthebroaderissueofhowtosettleongoals.Nordowediscusshowtorecognisewhenaresultofascientificstudy,formulatedusingconceptsthatcanbetackledwiththeproceduresofthestudy,isrelevanttothemoreabstractlyandvaguelysetoutgoalsthatareoftentherealaimsofpolicy.1Nordowepresentideashereonhowtocomeupwithasetofcandidatepoliciesforachievingagivengoal.Itisalsoimportanttokeepinmindthatwhetherapolicywillachieveitsstatedgoalsisonlyoneofmanyconsiderationsthatshouldgointopolicydecisions.2Wetreathereonlythefarsimplerbutalreadydifficultproblemofjudgingwhetheraparticularproposedpolicyislikelytoachieveaparticularalreadywellarticulatedgoal.

Oursecondstartingassumptionisthattheprojectneedstobeapproachedfromthepointofviewoftheevidenceuser,nottheevidenceproducer.

Third,weassumethatrigourisagoodthing,sothattheadviceshouldbefirmlyrootedinsoundprinciples;butwemustnotbepseudo-rationalistic.Arigorousargumentwithninewell-groundedpremisesandoneweakonedoesnotmakeforarigorouslyestablishedconclusion.Forthemostpart,estimates(p.293) ofwhetherapolicywillbesuccessfulmadeinrealtimewillbebothroughanduncertain.Thatisimportanttokeepinmindaspolicydecisionsaremade.Butitisalsoimportanttokeepitinmindasadviceguidesaredevised.Ifadviceistobepracticable,itmaynotbehugelyreliable,evenifitisultimatelywell-grounded.Weshouldaimforadvicethatimprovesdecisionsevenifwecannotdothejobperfectly.Thebestshouldnotbetheenemyofthegood.

Fourth,andcloselyconnectedwiththethird,isthatweshouldnotexpectpolicyeffectivenessjudgementstobeveryreliable.Thereareavarietyofdifferentreasonsconspiringtomakethesejudgementsespeciallydifficult,includingtheobviousdifficultiesofdoingwhatweproposehereasnecessaryforreasonablyreliablejudgements.Weshallnotrehearsethesereasonsbutjustofferoneremarktomakevividhowdifficultthetaskis.Askingifapolicyofaspecificdesignwillachieveatargetedresultisstructurallyjustlikeaskingwhetheralaserofaspecificdesignwillproduceacoherentbeamwhenweplugitin.Itisdifficulttoanswerthatquestionreliablybeforeactuallypluggingthelaserinitissimilarlycomplicatedtoproduceadviceaboutwhatcountsasevidencefororagainstananswerandabouthowtomarshalthatevidencetosettleonaprediction.Socialeffectivenesswillbeevenhardersincethesystemsunderstudyaremoreopen,ourtheoriesandknowledgeofthematerialsarelesssecure,andthechoiceoftargetedoutcomesisgenerallydictatedbysocialneed,notbyanassessmentofhowachievabletheyare.

HowtothinkabouttheproblemViewpointWhenitcomestoevidence-basedpolicy,viewpointmatters.Whetherwittinglyornot,


Page 4 of 30


typicaladviceguidesfocusontheproductionsideofscientificevidenceandnotontheuseside.Theytelluswhatcountsasgoodscience,nothowtousethatsciencetoarriveatgoodpolicy.

Mostavailableguides,liketheMarylandrules,theIARCschemeandWhatWorks,providerankingschemesforthequalityofevidence.Theseschemespolicethecredibilityofresultsthatcanbecountedasevidence.Evidenceclaimsarerankedaccordingtothemethodsbywhichtheyaretested.Highqualitymeansthatthetestsarestringent:Resultsthatpassthetestsareverylikelytobetrue.Randomizedcontrolledtrials(RCTs)arenecessaryforstrongevidenceaccordingtothedominantguides.Manyobjectonthegroundsthatthiscanmeanthrowingoutalotofgoodevidencethatweoughttobeattendingto.Thisissueisnotourconcernhere.Thecentralconcernweraiseisthat(p.294) theserankingsfocusontoonarrowarangeofclaimsthatneedevidencing,notthatthekindsofevidenceadmittedaretoonarrow.Why?

Truthisagoodthing.Butitdoesnttakeoneveryfar.Supposewehaveatourdisposaltheentireencyclopaediaofunifiedsciencecontainingallthetrueclaimsthereare.Whichfactsfromtheencyclopaediadowebringtothetableforpolicydeliberation?Amongallthetruefacts,wewantonthetableasevidenceonlythosethatarerelevanttothepolicy.Andgivenacollectionofrelevanttruefactswewanttoknowhowtoassesswhetherthepolicywillbeeffectiveinlightofthem.Howarewesupposedtomakethesedecisions?Thatistheproblemfromtheuserspointofviewandthatistheproblemoffocushere.

HereishowDrSeanTunis,directoroftheCenterforMedicalTechnologyPolicy,aUSorganisationconcernedwithwaystogetbettermedicalevidence,putstheproblem:TheresthisgulfbetweenwhatquestionsresearchershavefoundinterestingtostudyandwhatquestionsindustryandtheN.I.H.havechosentofundandwhatusersofinformationmostwanttoknow.Inourterms,thefocushasbeenonthesideofevidenceproduction,ratherthanevidenceuse:Onestartsfromtheheadandtheotherstartsfromthetailandtheydontmeetinthemiddle(citedinKolata,2008).

EffectivenessThereareagreatmanythingsweneedtoevaluateinconsideringwhethertoadoptapolicyornot.Willthepolicywork?Doesithaveunpleasantsideeffects?Doesithavebeneficialsideeffects?Howmuchdoesitcost?Havewemadethecorrectchoiceoftargetoutcomes?Isthepolicymorally,politicallyandculturallyacceptable?Canwegetthenecessaryagreementtogetitenacted?Dowehavetheresourcestoimplementit?Willenemiesoftheprojectsabotageitinvariousways?Everyoneofthesequestionsneedsansweringandineachcaseevidencewillhelpgettherightanswer.

Weshallconfineourdiscussion,however,tothequestionofeffectiveness:

QuestionofEffectiveness.Willtheproposedpolicyproducethetargetedoutcomeswereittobeimplementedinthetargetedsettingandimplementedinthewayitwouldinfactbeimplementedthere?3


Page 5 of 30


(p.295) AstructurefortheproblemStartthenfromthepointofviewofthepolicydeliberatortryingtoestimatewhetheraproposedpolicywillbeeffective.Forareliabledecisiononewantscredibleevidencethat,alltold,speaksfor(oragainst)thepolicy.Thissimpleobservationsuggeststhatfromthepointofviewoftheuserthreedifferentissuesneedaddressing:

1Quality:Whenareevidenceclaimscredible?2Relevance:Whendoesanestablishedresultbearonapolicypredictionandhowdoesitdoso?43Evaluation:Howshouldpredictionsaboutpolicyeffectivenessbeevaluatedinthelightofalltheevidence?

Thefirstisanissueabouttheproductionofknowledgebythesocialandnaturalsciences;itisthemeatofevidence-rankingsystems.Thelattertwoarethemoreneglectedquestionswefocuson.

Thefactthatthethreequestionsaredistinctshouldnotsuggestthattheiranswersareunrelated.Despitethecommonemphasisonquestion1,itseemsprimafacieasifthenaturalstartingpointiswithquestion2.Firstestablishwhatkindsofevidencearerelevanttoeffectiveness.Then,forquestion1,provideguidelinesthatpolicethequalityofevidenceofthosekinds;andforquestion3,proposesomeschemeforamalgamatingorcombiningevidence.

Inaidofthisapproachonecouldadoptoneoranotherofthecharacterisationsofrelevanceonofferfromphilosophyandmethodologyofscience,wherethetopichasbeenexploredanddebatedforyears;thenfollowonwithoneoranotheroftheschemesavailableforcombiningevidenceoradaptweighingschemeswithknowncharacteristicsfromotherareas,likethoseforamalgamatingpreferencesorexperttestimony.

Weadoptadifferentstrategy.Weproposetostartwithanaccountofhowtoevaluateclaimsofeffectivenessandworkbackwardstofigureoutwhatkindsofevidencewouldberelevantfortheevaluation,finallyreturningtothefirstissueofhowtoassurethatthekindsofevidenceclaimsneededaresufficientlycredibletoenterintodeliberation.

Beforebeginningwiththisaccount,wewanttostresstheimportanceforthesuccessofevidence-basedpolicyofcoveringallthreequestions.Question1isaquestionforknowledgeproducers:Whatisnecessaryinordertoensurethataclaimenteredasevidenceislikelytobetrue?Usershaveinadditionto(p.296) facequestions2and3.5Yetmostoftherigourandmostoftheattentionistoquestion1.Weareurgedtoextremerigouratonestage,thenlefttowingitfortherest.

But:achainofdefensefortheeffectivenessofapolicy,likeatowingchain,isonlyasstrongasitsweakestlink.Sotheinvestmentinrigourforonelinkwhiletheothersarelefttochanceisapttobeawaste.Tobuildtheentirechainonemayhavetoignoresomeissuesormakeheroicassumptionsaboutthem.Butthatshoulddramaticallyweakenthedegreeofconfidenceinthefinalassessment.Rigourisntcontagiousfromlinktolink.If


Page 6 of 30


youwantareasonablysecureconclusioncomingout,youdbetterbecarefulthateachpremiseissecureenoughgoingin.

II.Evaluatingeffectiveness

HowphilosophycanhelpWeproposetoborrowourthreecentralprinciplesofthetheoryofevidenceforusefromphilosophy.Thefirsttwoprovidethebasisofthetheoryandthethird,somepracticalhelpinimplementingit.

Truthvaluesforcausalcounterfactualsarefixedbycausalmodels.Causes,asJ.L.Mackieexplains,areINUSconditions.Inunderstandinghowcausesoperateandhowtheyoperatetogether,mechanismsmatter.

CausesandcounterfactualsForsoundpolicyweneedtoevaluatewhether,iftheproposedpolicywereimplementedasitwouldinfactbeimplemented,thetargetedoutcomewould(p.297) occurinconsequence.Wearelookingfortheprobabilityofwhatincausaldecisiontheoryiscalledacausalcounterfactual.6

Thereisgoodreasontoexpectanintimateconnectionbetweencausesandthesespecialkindsofcounterfactuals.Natureforgesit.Consider:Howdoesnaturedecidewhateffectstoproduceinaparticularsituation?Firstshesurveysthecausesthatwillbeoperating.Nextsheconsultsherrulesofcombinationtocalculatewhatshouldhappenwhentheyallactatonce.Thensheproducestheprescribedeffects.Wecantlosebyimitatingnature.

Thatisourproposal.Topredictwhatwillresultifweintroducesomenewpolicyorprogramme,followNatureslead:ReconstructNatureslistofcausesandmimicNaturescalculation.7Thisprovidesuswithagoodwaytopredicttheeffectsofourpolicyimplementationsandwecantgowrongifwesucceed.Moreover,anymethodthatdoesnotdirectlymimicNaturesprocesseswillonlygetpredictionsaboutcausalcounterfactualsright(orrightenough)ifithassomewayofachievingjustthesameresults.Later(inPartIV)weconsidercheapheuristicsthatmightgetthesameconclusionsenoughofthetimeinspecifickindsofcircumstances.Thesearegreatwhentheyareavailable.ButtheirconclusionsareonlywarrantedtotheextentthatwehavegoodreasontobelievethattheywillproducenearenoughthesameresultsaswouldacausalmodelthatmimicsNaturesprocedures.

Sinceitisoftennotpossibletomakelifeeasierandgettingthecausalmodelrightenoughisusuallyverydifficult,anyreasonablycomprehensiveguidewillalsoneedtoremindpolicyanalyststoexpectagreatdealofuncertaintyandtoadoptstrategiesfordealingwithitstrategieslikenotintroducingbigpolicychangesthataredifficulttoreverseandadoptingamuddlingthroughratherthangrandplanningapproach.8


Page 7 of 30


(p.298) CausalmodelsWeproposethentoadoptstandardphilosophicadviceasthefirstprincipleofthetheoryofuse:Toevaluatecausalcounterfactuals,buildacausalmodel(seeReiss,2007).Butthetermcausalmodelshouldnotcarryalotofbaggagewithit,eitherfromphilosophyorfromthesciences,wherevariousdifferentkindsofspecialisedcausalmodelsareonoffer.

Whatisacausalmodel?Forourpurposesacausalmodelhastwoessentialingredients,whereweseparatethefirstintotwopartstohighlightissuesaboutimplementationthatweknowpolicymakersneedtotakeintoconsideration.

1Alistofthecausesrelevanttothetargetedeffectthatwilloperateinthetargetsituation.Thisincludes

1.athecausespresentinthesituationindependentofthepolicyaction1.banychangesinthissetofcausesintroducedinimplementingthepolicy.9

2Aruleofcombinationthatcalculateswhatshouldhappenvis--visthetargetedeffectwhenthosecausesoperatetogether.

Considerasimplecase.Laterweshalllookatbothsomerealandsomepastichesocialpolicycases.Butfornowweillustrateusingeverydayphysics.Wedosobecausethereasoningissimple,well-understood,andwearenotlikelytogetinvolvedinsubject-specificdebatesineducationorcriminologyorhealthpolicy.Moreimportantly,wechoosethiskindofcasetostartwithbecauseitisonewhereourknowledgeoftheprinciplesandoftheaptnessoftheconceptsissecuresothatwecanfocusonthestructureofthereasoningneeded.

Thecaseofthedeskmagnetversustheindustrialmagnet.Wehaveaccesstoadeskmagnetandtoalargeindustrialmagnet.WeknowtheexactstrengthsofthesewithaveryhighdegreeofcertaintyclaimsabouttheirefficacyforliftingobjectshavepassedfarmorethantwogoodRCTs;theyhavecenturiesofstudybehindthem.Shallweuseoneofthemtoliftanobjectinmydriveway?Thatdependsontheotherfeaturesofthetargetsituation.

First,magnetsneedhelpingfactorstobeeffectiveatall.Adeskmagnetisuselessforliftingamatchstick;itisonlythecombinationofamagnetanda(p.299)ferrousobjectthatproducesamagneticforce.Thentheaccelerationcausedbythemagnetisstillonlyonepartofthestory,oftenoneverysmallpart.Toknowwhathappenswhenweapplythemagnetweneedtoknowtheotherforcesaswell.Here,especiallygravity.Thedeskmagnetmayliftapinbutitishopelessforacar,whereweneedtheindustrialmagnet.Wealsoneedtoattendtowhatotherforcesweintroduceinthecourseofgettingthemagnetinplace.Perhapstheindustrialmagnetwouldhaveliftedthecarifonlywehadntthrowntheheavypackingcaseforthemagnetintotheboot.


Page 8 of 30


Finallyweneedtoknowhowallthesefactorscombinetoproducearesult.Ofteninsocialcontextsadditivityisassumed:Addagoodthingandtheresultscanonlygetbetter.Butthatdoesntworkineventhissimplephysicalcase.Wegetsousedtovectoradditionthatweforgetthatitisntsimple(scalar)additionofeffectsizes.Addamagneticaccelerationof42ft/sec/sectothatofgravitys32ft/sec/secandyouwontusuallygetanaccelerationof74ft/sec/sec.

Thepointisthatwhetherthemagnetwillbeeffectiveatallinthetargetsituationandtowhatextentdependsonnaturescausalmodelofthesituation.Sothemostdirectwayofpredictingitseffectsistoconstructourowncausalmodelinimitationofnature.

Weknownoonewantstohearthissinceitseemsdifficult.Butconsider:IndustrialmagnetswouldpassanynumberofRCTs,ofanydegreeofstringency.Butthatsnotanywherenearenoughtoknow.Noneofuswouldrentanindustrialmagnettoremovealoadofrubbishwithoutlookingattherubbish.Knowledgethatmagnetsjustlikethiscanliftisonlyasmallpartofwhatonewouldconsiderinevaluatingwhetherrentingtheindustrialmagnetwillbeeffectiveinremovingourrubbish.Ifthisissoineverydaycalculationsandinappliedscienceandengineering,whyshouldweexpectittobesubstantiallydifferentandsubstantiallyeasierinsocialengineering?

Ofcourseconstructingcausalmodelsishard,evenifthemodelsareroughandwehavefiguredoutwaystotolerateuncertainties.Sometimesthereareshortcuts,cheapheuristicsthatgetus,more-or-less,well-enough,thesameconclusionsthatthecausalmodelgenerates.Asdecisionmakerswecanoptforaheuristicifwewant.Butthereisnoavoidingthefactthatthechoiceoftherightheuristicdependsontherightcausalmodel.Wemaynotwishtobuildacausalmodel;wemaynotknowhowto;wemaythinkittakestoomuchtimeormoney,intelligenceorattention.Thatdoesnotalterthefactthatwhenwebuyapolicywearebettingonacausalmodel,willy-nilly,whetherwewishtothinkaboutitornot.

HadweworldenoughandtimeAgreatdealmorecanbesaidaboutcausalmodels.Butitissubject-anddiscipline-specificandalmostalwaysrequiresexpertiseandtrainingtodoat(p.300) allproperly.Moreover,manyscientificmodelsdolessthanwhatwedemandofacausalmodel,thoughtheyprovidemoredetailandzeroin,usuallyveryprecisely,onspecificfeaturesofinterest.

Considerajointefforttoexplorethecausesofdelaysinemergencyrooms(Lane,MonefeldtandRosenhead,2000).ThemodellingexpertisewasprovidedbytheDepartmentofOperationalResearchatLSE,whileorientationtotheproblemarea,judgementsondesignchoices,andintroductionstostakeholdersweresuppliedbyCasualtyWatch,aprojectorganisedasaresponsetopublicconcernthatcutsintheNHSwereproducinganinadequateemergencyserviceandharmingpatients.SystemdynamicswasselectedastheappropriatemodellingmediumandthemodelwascalibratedwithinformationfromaninnerLondonteachinghospital.


Page 9 of 30


Hereswhatthemodellookslike:

Figure11.1 .Modelofdelaysinemergencyrooms.

Whatsimportantaboutthismodelisitsabilitytodetectandrepresentfeedbackloopsanditsdynamicstructure.ForinstanceitmakesclearthatthenumberofbedsavailableinthewardsbothaffectsandisaffectedbythenumberofadmissionsfromA&EandthatthenumberofpatientsbeingtendedinA&EaffectsandisaffectedbythenumberofpatientsbeingadmittedtothewardsfromA&E.Italsoshowsanumberofpathwaysbywhichaninitial(p.301) cause,sayarrivalsattheAccidentandEmergencyDepartment,influencesthefinaleffect,patientwaitingtimeatA&E.

AsweshallexplaininPartV,tracingthroughthedynamicslikethis,step-by-step,canbeabighelpinconstructingasignificantpartofthesecondcomponentwedemandinacausalmodelanaccountofhowcausesacttogethertoproducethetargetedeffectbecauseitfocusesonwhatauxiliarycausesareneededateachstepifthesalientcauseistoproducethenextstepintheprocess.Notice,however,thatthisinformationisnotexplicitlyrepresentedinthemodelsincethemodeltreatscausessingly.Attheheadofthearrowatthecausesendisasinglevariable;e.g.bedcapacity,warddischargerate,andemergencyadmissionrateareallpicturedasseparatecausesofthewardoccupancyrate.Thereisnoinformationencodedabouthowthesedifferentcausescombine,inparticularwhichcausesmustacttogetherbeforetheycancontributetotheeffectatall.Thusthismodel,likemostprofessionalmodels,doeslessthanwerequire,thoughwhatitdoes,itdoesmorepreciselyandinmoredetail.

Anotherexample(Figure11.2below),thisfromJudeaPearl(1995:66970).


Page 10 of 30


Figure11.2 .AcausalBayesnet.Variables:X:fumigants;Y:yields;B:thepopulationofbirdsandotherpredators;Z0:lastyearseelwormpopulation;Z1:eelwormpopulationbeforetreatment;Z2:eelwormpopulationaftertreatment;Z3:eelwormpopulationattheendoftheseason.

(p.302) Inthismodel,asinthelast,causesareatthetopofthearrow,effectsatthetip.BycallingitacausalBayesnetspecialassumptionsaremadeabouttherelationsamongthevariablesthatmaynotholdineverycausalmodel;forinstancecausesandeffectspicturedinthegraphareallsupposedtobeprobabilisticallydependent.Generallythiskindofmodelcomeswithnumbersaswell,ideallytheconditionalprobabilityforeacheffectconditionalonalltheimmediatelypriorcausesleadingintoit.Sothesemodelscontainmoreinformationthanisrequiredbyourtwoconditionsforacausalmodel,informationofspecialuse10intheparticularkindsofcausalsystemsthatsatisfythespecialaxiomsthatrelatecausesandprobabilitiesinaBayesnet.11Butlikethedynamic-systemsmodelforemergencyroomadmissionsandhospitalbeds,italsocontainslesssincethemodeldoesnotshowhowthecausesinteractamongthemselvestoaffectyields.WeknowfromthegraphthatZ2caninfluenceYbutevenifweaddtothatknowledgeoftheconditionalprobabilityofYonZ2wedontknowfromthegraphwhetherthepresenceorabsenceofXisessentialtotheabilityofZ2toinfluenceY.12

Thiskindofmissinginformationisreadilysuppliedbymodelspresentedintheformofequations,iftheycanbeconstructed.HereforinstanceisthefinalequationfromacausalmodelweshalldiscussinPartV:

(p.303)

(*) = [ ] + y +t t t1 pt t


Page 11 of 30


Hereytisoutputattandptispriceattso[ptpt1]isameasureofinflation;etisarandomerrorvariable.13ThisequationyieldsasanextsteptheclassicPhilipscurverepresentingatrade-offinwhichrisinginflationcausesdecreasingunemployment.Oncetheparameters,,,and,arefilledintheequationshowshowthetwocausesrepresentedinflation,[ptpt1],andearlieroutput,yptcombinetoproducelateroutput,yt:inthiscase,simplelinearaddition.14

Wewillpresentasimplephysicsexample(seeIllustrationsofINUSconditions,below,p.305)whereacompletesetofcausesisalsolaidoutinanequation,buttherulesofcombinationforthecausesaremorecomplicatedinvolvingnotsimple(scalar)additionbutalsomultiplicationandvectoraddition.

Equationsforcalculatingtheexactresultofagivensetofcausesarewonderfulwhenonecangetthem.Buttheymaynotbepossibleeveninprincipleformanycases.Evenacompletesetofcausesmayactonlyprobabilistically,notfixingavaluefortheeffectatallbutonlyaprobability.Infactwehazardthatthatismoreoftenthecasethannot.Andeventhatmaybewishfulthinking.Natureherselfmayproceedwithlessquantitativeprecision,notfixingevenafinalprobability,perhapsonlyadirectionofchange.Whethershedoessoornot,thislevelofprecisionisgenerallywellbeyondtheabilityofnormalpolicydeliberators.Also,asourcolleaguesatarecentconferenceoncausalityurged:Ourlistofcauseswillalmostalwaysbeincomplete;theverybestwecanhopeforisaprobabilisticassessmentoftheoutcomesandeventhatshouldgenerallynotbetooprecise.Sodontgethunguptryingtoproduceequations.

Butthatisnotadvicetoignoretheneedtogetagriponthedominantcausesthatwillaffecttheoutcomeortheneedtobetonwhattheydoincombination.Itisjustadvicenottoexpectadegreeofprecisionoradegreeofconfidencethatneitherthesubjectnorourcapabilitiescansupport.

(p.304) INUSconditionsIntroductionToevaluateacausalcounterfactualoneneedstoconsiderthemajorcausesatworkandhowtheycombine.Onecharacteristicofcauseswidelyacceptedinphilosophycanhelpwithbothenterprises.AsJ.L.Mackie(1965)argued,causesareINUSconditions.15

AnINUSconditionisanInsufficientbutNecessarypartofanUnnecessarybutSufficientcondition.

Thefactorswenormallycallcausesare,accordingtoMackie,INUSconditions.Causesinourusualsenseofthewordarenotenoughontheirowntoproduceaneffect.Causesworkincooperation;theyneedhelpingfactors.Ittakesbothalightedmatchandagoodstackoflogsandbrushtoproduceabonfire.TogetherasetoffactorsthatareSUFFICIENTtoproduceaneffectmakeupwhatweshallcallacompletecausalcomplex.Eachfactorinthecompletecausalcomplexforexample,thebrushorthelogs

(*) = [ ] + y +yt pt pt1 pt t


Page 12 of 30


orthelightedmatchisINSUFFICIENTbyitselftoproducetheeffect.Still,eachhasgottobethereorthatcomplexwontproducetheeffect.ThatswhytheseparatefactorsinthecomplexareNECESSARY.

Thewholecomplexitselfhowever,whilesufficientfortheeffecttooccur,isgenerallyUNNECESSARY.Thatsbecausetherearealmostalwaysotherwaysothercompletecausalcomplexestoproducethesameresult:Onecanalsomakeabonfirewithastackofdrystrawandpackingcasesandacigarettelighter,orwithdrystraw,packingcasesandawell-aimedboltoflightening.Eachofthesedifferentcompletecausalcomplexesissufficienttomaketheeffectoccurbutnoneisnecessarysinceeachoftheothercompletecomplexeswilldoaswell.Andeachcomplexcontainsanumberofcooperatingfactors,likethelighterorthebrush,eachoneofwhichisinsufficientbyitselffortheeffectbutisnecessaryifthecomplexofwhichitispartistodothejob.

INUSconditionsarenotjustatopicforphilosophers.EpidemiologistshavedevelopedacompellingwaytounderstandINUSconditionswiththeuseofpiegraphstorepresentsufficientandcomponentcauses.Eachsliceinagivenpierepresentsacomponentcauseandawholepierepresentsasufficientcause,acompletecausalcomplex.Asinglepiesliceonitsownisinsufficienttocauseadisease;thewholepieisneeded.SointhephilosophersvocabularyasinglepiesliceisanINUScondition.

(p.305) Belowaretwocompletecausalcomplexesforadiseasewiththecomponentcausesshownaspieslices.Therearesomesharedcomponentcauses(C1andC2)andsomeuniquecomponentcauses(C4andC8,forexample).Also,weindicatetheunknowncomponentcausesasCNintheleftpieandCMintherightpie.

Hereisanexample.Smokingcauseslungcancerbutnotallsmokersdeveloplungcancer.Thereareotherfactors,perhapsgeneticfactorsandotherlife-styleandenvironmentalfactors,thatcontributetodevelopinglungcancer.SoinFigure11.3,SufficientCauseAwouldbetheconstellationoffactors,includingsmoking,thattogethercauselungcancer;smokingcouldbeC3.Butpeoplealsodeveloplungcancerwithouteversmoking.SoinFigure11.3,SufficientCauseBwouldbeaconstellationoffactorsnotincludingsmoking(C3isnotpresent)thattogethercauselungcancer.WorkinginacoalmineforexamplecouldbeC8.

Figure11.3 .Twosufficientcausesandtheircomponentcauses.

IllustrationsofINUSconditions


Page 13 of 30


InthissectionweprovideexamplesfromdifferentsubjectstoillustratewhatINUSconditionsareandhowtheyworktogethertoproduceaneffect.Thefirstisanexampleabouttheeffectivenessoflawsrequiringbicyclehelmetsinreducingheadinjuriesamongcyclists.

BICYCLEHELMETEXAMPLEVigorousdebateregardingtheefficacyofbicyclehelmetstoreduceheadinjuryhasbeenpublishedinthepagesoftheBritishMedicalJournal(seeespeciallyRobinson,2006:7225andnumerouslettersinresponse).Casecontrolstudiessuggestthatcyclistswearinghelmetshavefewerheadinjuriesthancyclistsnotwearinghelmets,whereastime-seriesstudiesinjurisdictions(p.306) thathavepassedhelmetlawsdonotshowacleardecreaseintherateofheadinjuriesafterhelmetlawshavebeenimplementedandinsomecasesthesestudiessuggestanincreaseinheadinjuriesafterthelawisimplemented.

Atfirstglancethisisparadoxical.Ourintuitions,supportedwithevidencefromcasecontrolstudies,saythathelmetsshouldreduceheadinjurieswhereashelmetcompulsionlawsfailtoshowmuchbenefitandinsomecasespossiblyshowanincreaseinheadinjuries.

Therearemethodologicalreasonsthatcouldpartlyexplainthedifferencesbetweenthesestudies.Aworryaboutconfoundersinthecasecontrolstudiescouldexaggeratetheestimatedefficacyofhelmets:Thereissomeevidencesuggestingthathelmetwearersareoverallsaferbicycleriders,areinvolvedinlesssevereaccidents,arericher,andmorelikelytobewhite.Aworryaboutconfoundersinthetime-seriesstudiescoulddampentheresultofintroducinghelmetlaws:Overtheperiodsofthesestudiestherehavebeenmorecarsonroadsandthesecarshaveincreasedinsizeandspeed.

Leavingasideadiscussionofthemethodologicalqualityofcasecontrolstudiesversustime-seriesanalyses,thisparadoxcanbeunderstoodbythinkingaboutINUSconditions.Thecasecontrolstudiesgiveonepieceofacausalpie:Helmetscancauseareductioninheadinjuries.Butthosestudiesdonttellabouttheotherpiecesofthepie,thatis,otherfactorsthatarecausallyrelevanttoacyclistsheadinjury,thingslikedriverbehaviour,cyclistbehaviour,androadconditions.Now,thereisevidencetosuggestthatatleastsomeofthesethingschangewithhelmetwearing.16Driversgivelessspacetocyclistswhoarewearingahelmetandcycliststakemorerisks(afalsesenseofsecurityphenomenon).Sohelmetcompulsionlawsdontjustchangeonepieceofacausalpie,theychangeseveralpieces.Andthatcouldpartlyexplainthediscordancebetweenthetwokindsofstudies.

Thenicethingaboutthisbicycleexampleisthatitillustratestwolessonsatonce.First,theimportanceofidentifyingtheotherINUSconditionsthatgointoacompletecausalcomplex,i.e.theotherslicesinthesamepiewhichonecanthinkofashelpingfactorsnecessaryinorderforthepolicylevertowork:Helmetwearingincombinationwithusualdriverbehaviourwilldecreaseheadinjuriesfrombicycleaccidents;helmetwearingwith


Page 14 of 30


moredangerousdrivingmayincreaseheadinjuries.

Second,itremindsusthatinthinkingaboutINUSconditionsweneedtopayattentiontotheunintendedconsequencesofouractions.Inimplementing(p.307) apolicywemaynotonlyproduceunwantedsideeffects;wecan,asinthiscaseandintheLucasexampletobediscussed,introducefactorsthatunderminetheeffectivenessoftheverypolicyleverweemploy.Ofcoursewewillalwaysbeplaguedbyuncertainty.Weareinnopositiontopredictmanyoftheunintendedoutcomesofourpolicies.Butsomewecanpredictifonlywethinkaboutthemintherightway.

ThefailureoftheCaliforniaclass-sizereductionprogrammemaywellbeacaseinpoint.Thereductioninclass-sizewasrolledoutstate-wideoveraveryshorttime.Thatnecessitatedthehurriedhireofalargenumberofnewteachersandinconsequenceteachingqualitywentdown(BohrnstedtandStecher,2002).Butteachingqualityisasliceofthesamepieassmallclasssize:Reducingclasssizecannotbeexpectedtoincreasereadingscoreswithoutthecooperationofgoodteaching.Thepointisthatthisunintendedconsequenceofthepolicyimplementationisthekindthatmightwellbeforetoldifcarefulthoughtisputtowardsit.Soinproducingapracticableguidebasedontheprincipleshere,onewillhavetofigureoutwaystoreminduserstothinkabouttheunintendedconsequencesoftheirpoliciesandimplementationsandtohelpthemdoso.

PHYSICSEXAMPLEAnobjectofchargeq1withcentreofmassatadistancerfromtheearthscentreisacceleratingatadistancerfromasecondobjectofchargeq2.Itisalso,ofcourse,subjecttotheearthspull.LettingMrepresentthemassoftheearth,theobjectsaccelerationisgivenby:17

Thefirstterm(theCoulombacceleration, q1q2/r2)issufficientitisenoughtoobtainacontributiontotheacceleration.Butitisunnecessary.TherearealotofothercausesthatcancontributetotheaccelerationeveniftheCoulombforceisntthere.Sotoowiththesecondterm(theaccelerationduetogravity,GM/r2):Thepresenceoftheearthsmassadistancerawayissufficientforproducingacontributiontotheaccelerationbutitisnotnecessary.

Considernextq1.WithoutitthereisnoCoulombforce.Soitisanecessarypartofthefirstterm.Butitisinsufficientsinceitcannotproduceacontributiontotheaccelerationonitsownbutonlyinconsortwithanothercharge(q2)(p.308) andsomeseparation(r).Thesameistrueofeachoftheotherfactorsappearinginthefirstterm,aswellasofthefactorsMandrinthesecondterm.

Thefactorsq1,q2,r,M,andrareallcausesoftheaccelerationinanybodysbooks.Andtheyareeach,asMackieclaims,INUSconditions;eachisinsufficientbutnecessarytoacausalcomplexthatissufficientforobtainingacontributiontotheacceleration,butnooneofthesesufficientcausalcomplexesisnecessaryforacontribution.Moreover,we

Acc = / +GM/rq1q2 r2 2


Page 15 of 30


knowthefunctionalformoftherelationbetweenthefactors.

FunctionalformMerelyknowingwhattheINUSconditionsareislesshelpfulthanknowingtheformalrelationshipamongthefactorsbutweneedtoknowwhatthefactorsarebeforewecaninvestigatetheirrelations.Inthephysicsexampleweknowthefullfunctionalformfortheproductionofacceleration:Weknowallthepossiblecauses;weknowwhichonescombinetogethertomakeasinglecomplexsufficientforproducingacontribution;weknowthefunctionalformfortheirmutualrelationswithinthecomplexese.g.thedistanceintheCoulombtermappearsinthedenominatorandissquared;andweknowhowthecontributionscombinetoproduceanoveralleffectbyvectoraddition.

Therearestandardmethodsusedinthesocialsciences,andespeciallyineconometrics,forteasingoutaspectsofthefullfunctionalformoftherelationsbetweencausesandeffects,andclearlyphysicshasbeenverysuccessfulatthis.Thatsidealforpredictingcausalcounterfactuals.Mostoftenforrealpolicycasesinrealtime,however,thereislittlehopeformuchheadwayonthefullfunctionalform.ThatiswhywehaveoptedtofocusonINUScontributionsatleastwithareasonableunderstandingoftheseonewillknowwhatauxiliariesarenecessaryifthepolicyvariableistohaveahopeofbeingeffective.Butitisworthhavingtheidealinmindsinceitisstructurallylikethelessidealcasesthatmustbedealtwithinsocialpolicy.

SomephilosophicalnicetiesDICHOTOMOUSVARIABLESVERSUSCONTRIBUTIONSMackieintroducedINUSconditionsinthecontextofdichotomousvariables,thatis,avariablethattakesyes/novalues.Doesthepatientsurvive;doesthemagnetliftthepin;doesthebicyclistsustainaheadinjury?ForMackieacompletecauseisasufficientconditionforaneffectinthelogicianssenseofsufficient:Thepresenceofthecompletecauseimpliesthepresenceoftheeffect.Inthiscasethereisnoquestionofhowdifferentcompletecausescombine.(p.309) IfCimpliesEthenCandCimpliesE,nomatterwhatCis.Soifanyonesufficientconditionforaneffectispresent,theeffectispresent;addingmoremakesnodifference.

Manyoftheeffectsofinterestinsocialpolicyarenotdichotomoushoweverbutcantakeavarietyofvalues,likeaccelerationinoursecondexample.Thatis,thevariablesofinterestaremulti-valuedratherthandichotomous.Inthesecaseseachcompletecausalcomplexoperatingonitsownwillproducesomevaluefortheeffect.Butwhentheyacttogethertheeffectwillbedifferentfromthatproducedbyanyonealone.Eachaffectsthevalueoftheoutcomebutdoesnotdetermineit.Whenthishappens,onecantalkaboutthecontributionthecompletecausalcomplexmakestotheeffect,aswedidinthephysicsexample.Thenthepossibilitiesfortherulesabouthowcausescombinemultiply.Themostobviousaresimpleadditionandsubtraction.Buttherearemanyotherpossibilities,asinthevectoradditionofmechanicsorloglinearcombinationprevalentineconomics.Thesearetherulesneededforthesecondcomponentofanidealcausalmodelofthekindweurgeabove(Whatisacausalmodel?,p.298).


Page 16 of 30


Nowthatwehavemadeexplicitthedifferencebetweenhowcausesworktogetherinthecaseofdichotomousversusmulti-valuedeffects,itistimetotidyupanearlierformulation.WeurgeCausesareINUSconditions.ButwedidnotsayforwhattheyareINUSconditionsthoughourlanguageindiscussingtheexamplesrevealsthattherearetwodifferentanswers.FordichotomouseffectsacauseisanINUSconditionfortheexistenceoftheeffect.FormultivaluedeffectscausesareINUSconditionsfortheexistenceofacontributiontotheeffect.InthephysicsexampleforinstancewhereboththeCoulombforceandtheforceofgravitycontributetotheacceleration,q1andq2arebothinsufficientbutnecessarypartsofacausalcomplex,q1q2/r2,thatisitselfsufficientforacontributiontoaccelerationbutnotnecessarysinceacontributiontoaccelerationcancomefromothersources,likegravity(formoreoncontributionsseeinteraliaCartwright,2007and2009).ThroughoutwewillcontinuetousetheexpressionINUSconditionforeffectXambiguouslytorefertoINUSconditionsforthepresenceofXwhenXisdichotomousandtorefertoINUSconditionsforacontributiontoXwhenXismulti-valued.

NOTALLINUSCONDITIONSARECAUSESCauses,wesay,areINUSconditions.Beware.Wedonotsay,INUSconditionsarecauses.Thereasonisthewell-knownproblemofspuriouscorrelation.Twofactorscanbecorrelatedwithouteithercausingtheother;similarlytwofactorscanbesufficientforeachotherwithouteithercausingtheother.(p.310) Considerasimplecaseofdichotomousvariables,whereonefactorCcausesbothEandE,neitherofwhichhasanyothercauses.ThenbothEandEoccurifandonlyifCoccurs,whichimpliesthatEoccursifandonlyifEoccurs.SoEandEareeachsufficientforeachother.SowedontclaimthatallINUSconditionsarecauses.ButweagreewithMackieandotherphilosophersthatcausesareINUSconditions,eitherfortheireffectsorforcontributionstotheeffects.

WhyfussaboutINUSconditions?Usuallywhendiscussingpolicyonefocusesonasinglecause,thatis,asingleINUScondition.ButitisnotpossibletopredicttheeffectofthatcausewithoutconsideringalltheotherINUSconditionsandtherelationsamongthem.18ThinkingintermsofINUSconditionsthenservesseveralpurposes:

Itfocusesattentiononthefactthatthereareusuallyanumberofdistinctcausalcomplexesthatcontributetotheeffect.(Soonedoesntexpectthematchtolightthelogswithoutthedrybrush.)Itfocusesattentionontheotherfactorsthatarenecessaryalongwiththepolicyvariableifthepolicyistohaveanyeffectatall.(Sowedontbothertorentthemagnetiftherubbishisntferrous.)Itfocusesattentiononthefunctionalformoftherelationsofthevariableswithinasinglecausalcomplex.(SoweexpectthatincreasingtheseparationbetweenchargesdoesnotincreasebutdecreasestheCoulombaccelerationbecausetheseparationisinthedenominator.)Itfocusesattentionontheoverallfunctionalform:Howdotheseparate


Page 17 of 30


causalcomplexescombine?(Recallourearlierremark.Ofteninsocialcontextsoneassumesadditivity.Butthatdoesntworkinevensimplephysicalcases.Thevectoradditionofclassicalmechanicsisafterallalongwayfromthesimplelinear(scalar)additionofeffectsizes.)

AndthenotionofINUScontributionisusefulbecauseitmoreadequatelyaccountsforthefactsthatmosteffectvariables,oroutcomesofinterest,arenotdichotomousandthatmostcausalfactorsthemselvescontributetotheeffecttovaryingdegreesratherthandichotomously.

(p.311) TwocentralprinciplesforatheoryofuseWenowhavetwoassumptionsthatformthecoreofatheoryofevidenceforpolicyeffectiveness:

Principle1:Agoodwaytoevaluatewhetherapolicywillbeeffectiveforatargetedoutcomeistoemployacausalmodelcomprising

AlistofcausesofthetargetedoutcomethatwillbeatworkwhenthepolicyisimplementedAruleforcalculatingtheresultanteffectwhenthesecausesoperatetogether.

Principle2:CausesareINUSconditions.

III.TheneglectedquestionsWiththesetwotheoreticalprinciplesinplacewereturntothethreeissuesofquality,relevanceandevaluation.Ifoneistoevaluatepolicycounterfactualsviacausalmodels,aswepropose,thisimposescriteriaofrelevanceandalsoaffectsthestandardsofquality.Acausalmodel,evenifroughandapproximate,requiresagreatdealmoreinformationthanweareinthehabitoflookingfor.

Requisiteinformationforevaluatingpolicyeffectiveness:Informationisneededabout:

Thecausalfactorsthatwilloperate:Whatfactorscausallyrelevanttothetargetedoutcomeareinthesituation?Thisbreaksnaturallyintotwoquestions:

Whatsthere?Isitcausallyrelevant?

Whatfactorsthatareintroducedduringimplementationwillbecausallyrelevant?Againthisbreaksintotwoquestions:

Whatwillwedo?Whatfactorsamongthoseweintroducewillbecausallyrelevant?

Howthesecombineinproducingtheeffect.Hereoneshouldpay


Page 18 of 30


particularattentiontoWhatauxiliaryfactorsarenecessaryalongwiththepolicyvariabletoproducethetargetedeffect?Howdodifferentfactorswithinasinglecomplex(differentsegmentsofthesamepie)combine?Howdodifferentcausalcomplexes(differentpies)combine?

Theseareempiricalquestionsandanyanswersthatareproposedshouldhaveevidencetosupportthem.Thissetsourcriterionofrelevance:

(p.312) Anempiricalclaimisevidentiallyrelevanttoapolicyeffectivenessestimatejustincaseithelpstoestablish:

i.Whatsthereinthetargetsituationii.Whatwillbeintroducedinimplementingthepolicyiii.Thecausalrelevanceofanyoftheabovefactorsforthetargetedeffectiv.Themethodofcalculatingjointeffects.

Thisformulationdoesnoteliminatequestionsofrelevance;itonlypushesthembackalevel.Onestillneedstoknowwhatkindsofevidencearerelevantforestablishingwhatsthere,whatfactorsarecausallyrelevant,andforclaimsofhowtheycombine.Thepointatthemomentisthatrelevanceisafarbroaderchurchthantheoneweareusedtopractisingin.Inprincipleoneshouldhaveevidenceforallthecomponentsthatneedtobeusedinsupportinganeffectivenessclaim.Inpracticesomefactswillbefairlyobviousandnotneedmuchevidencing;andonewillnecessarilytakeagoodmanyshortcuts.Butthetaskforthispaperisnottojumpintoshortcutsbutrathertolayaprincipledfoundationforjudgingpolicyeffectiveness,includingevaluatingshortcutsanddecidinghowmuchtobetonthem.

Thebroad-churchrelevancecriteriainturnaffectissuesofquality.Mostcurrentguidesfocusonthequalityofefficacyclaims.Dependingoncontextandphilosophicalleanings,thesecanbereadasclaimsthatthepolicycanwork,orthatitdoesworkunderspecificconditions,oraboutitsaverageeffectinaparticularpopulationunderspecialimplementationsacrosssomerangeofconditions.Efficacyclaimshelpsupportthecausalrelevanceofthepolicyvariable,whichispartofcategoryiii.Theusualrankingschemespolicethequalityofefficacyclaims.Buthowshouldthequalityoftheotherkindsofclaimsneededasevidencefortheremainderbepoliced?

Thisissueneedstobefacedanddealtwith,howeverfallibly,indesigningawell-groundedcomprehensiveadviceguide,convenientasitwouldbetoignoreit.Recallourcautionsaboutchainsofargument.Itisnousehavingoneortwohighlycertainpremisesinarguingfororagainstpolicyeffectiveness.Theconclusioncanbenomorecertainthantheweakestpremise.Inadoptingapolicy,oneisbetting,willy-nilly,thatalltherequisitequestionshavetherightkindsofanswers.Onecandothatonawingandaprayer.Butthatisnotanevidence-baseddecision.Soitisimportanttofigureoutreasonableandusablesetsofadviceabouthowtomanagetheneedforevidenceandnotto


Page 19 of 30


institutionaliseignoringtheneed.

HereisprobablywhereNancyCartwrightfirstgotintotroublewiththosewhomaintainthatRCT-backedpoliciesaretheonlyoneswithareasonableevidencebase.WearehappytotakeRCTsasagoldstandardforsomething.Inourviewtheyareprovablygoodatestablishingefficacyconclusions,asare(p.313) anumberofothermethods,suchasdeductionfromsoundtheoryandcertaineconometricmethods(SeeCausalClaims:WarrantingThemandUsingThem,inCartwright,2007).Butthatisfromthepointofviewoftheevidenceproducer.

Evidenceuserswanttoknowifapolicywillworkforthem.That,aseveryonehasreallyknownallalongandaswehavebeenstressinghere,requiresalotmoreinformationthantheinformationsuppliedbyanRCToragoodeconometricmodelthatestablishestheefficacyofthepolicyvariable;andthatinformationneedsevidence,includingevidenceaboutwhatcanoftenbeareallytoughquestionhowthecausescombine.

Thingslookverydifferentwhenonesurveystheproblemfromtheuserspointofviewfromhowtheydowhenlookedatfromthepointofviewofthescientistchargedwithproducingsoundresultstoofferupasevidence.19Imagineweareofferedtwopolicies.OnehasverygoodRCTevidenceinfavourofitsefficacybutwehaveveryweakideasandinformationaboutwhattherequisitehelpingfactorsandmajorinhibitorsforitare.Thesecondisapolicythatcomeswithatheorythatsuggestswhathelpingfactorsareneededandtheseareonesthatareeitherinplaceforusorcheaptoputinplace.Supposethetheoryhassomereasonableevidenceinitsfavourandtheassociatedpolicyhassomeevidenceforefficacy,butnotgoldstandard?Whichhasstrongerevidentialsupportinfavourofitsclaimtobeeffectiveifweimplementit?

Thisisaquestionthatdependsontheactualdetailsandinmanycasestherewontbeanyverygoodanswer.Butsometimesnormaleducatedjudgementwillandshouldreasonablygoforthesecondpolicythoughtheevidenceforitsefficacyisclearlylesscompelling.Thatswhywemadesuchanissueatthestartofthispaperaboutchainsofsupport,whichareonlyasstrongastheirweakestlink.Addingmorerigouratonepointcanraisetheoverallprobabilitythatthepolicywillbeeffectivebutthatincreaseinprobabilitycanbeoffsetbytoomuchguessinglateron.Wedonothaveguidesthatprovideenoughoftherightkindofadviceconsideringallthatisrequired.

(p.314) Itwouldbewrongofcoursetosuggestthattheseotherissueshavenotbeentackledatall.Alotofhardworkandseriousthoughthasbeenputintowhatisalreadyavailable.Butmuchofitispiecemeal,directedatspecificproblems,startingfromspecificplacesinmidflow.Weneedafoundationthatconsiderstheproblemofevaluatingeffectivenessofcounterfactualsasawhole.Itisonlyonthebasisofsuchafoundationthatonewillbeinapositiontojudgehowreasonableitistoleaveoutspecificconsiderations,totakespecificshortcutsandtomakespecificheroicassumptions.Thetheoreticalfoundationproposedhereismeanttodothatjob.Itisnottheonlyonepossiblebutitisafoundationlaidspecificallywithaviewthatpracticableadviceneedstobebuiltupfromit.


Page 20 of 30


IV.MakinglifesomewhateasierPerhapssuggestingthatwewanttoprovideanadviceguidebasedontheideaofconstructingacausalmodelsoundslikeatallorder.Sometimesitis,particularlywhenthereisademandforveryprecisepredictionsorpredictionsthatonecanbeverysureof.Butweshouldnotbetoofrightenedoftheproject.Foritisonewearewellusedto.Weregularlybuildcausalmodelsinmakingdecisionsinourdailylivesaswethinkthroughthepossibleeffectsofouractionsandpolicies.Consequentlytheschemashouldnotbeseenastooexoticorimpractical.It,orsomethinglikeit,isusedallthetime.

Forexample,recentlyNancysfavouritered-and-white-stripedteeshirtwassoiledlooking.Shouldshewashitinhotwater?Well:Hotwateronlyworksiftheshirthasareasonableamountofcottoninitanditwontworkagainstcoffeeorinkstains.Evenwithcottonitcanbecounterproductiveifthehotwatermakesthestripesrun.Andsheknowsthatshehastobeespeciallycarefulinloadingahotwashsincetheshirtwillgogreyifsomedarksocksareinadvertentlyincluded.Alltold,givenhercottonshirtwithgardendirtandthedeterminationtobecarefulinloadingthemachine,shereckoned(correctly)thattheshirtwouldcomeoutcleaninahotwaterwash.

Thisisahomelyexamplebutitillustratesourclaimthatpeoplebuildwhatwecallcausalmodelsallthetimewhenmakingpolicydecisions.Theproblemforevidence-basedpolicyishowtouseevidencetobuildthembetterandtoestimatethedegreeofconfidencepolicyanalystsshouldhaveintheresultsoftheirefforts.

Perhapsyoudonotfindthisfamiliarkindofexamplecomforting.Theideaofinsistingoncausalmodelsstillssoundstoodaunting.Nevertheless,Naturewilluseacausalmodeltodecidewhatoutcomestoproducewhenweimplement(p.315) ourpolicieswhetherwewishtofollowherleadornot.Therightanswerstothequestionsofqualityandrelevancewilldependonthemodelsshechooses.So,dauntingornot,adviceonthesequestionsshouldreflectthat.

Wecan,however,sometimesmakethejoblessdaunting.Consider:Wewouldingeneralliketobeabletopredicttheactualvalueoftheeffectthatwouldfollowtheimplementationofaproposedpolicy.Byjusthowmuchwillhouseholdburglariesdropifacommunity-widepropertymarkingprogramisadopted?Butoftenthatwillbedifficultbecausewedonotknowhowtopredictwhatelsewillbegoingon.Whatothercausesofburglarieswillbeinplaceatthetime?Oftenwecannotassumethatthecauseswillbethesamethenastheyarenow.(ThisisthereasonJ.S.Millsaideconomicscannotbeaninductivescience.)Sowecantestimatewhatothersufficientcausalcomplexeswillbeatworkcontributingtotheoutcome,letalonewhattheircombinedeffectwillbe.Inthesecaseswemaybesatisfiedwithreasonableassurancethatthepolicywillproduceanimprovementintheeffectoverwhatwouldbethecasewithoutit,whateverthatis.Ifso,lifeissomewhateasier.

Inthiscaseestablishingjustacoupleoffactswillallowustoignoretheothersufficientcausalcomplexes(alltheotherpies)andconcentrateonthosethatincludethepolicyvariable.20Whatweneedtoknowisthatnoalternativecomplexofcauseswillbeso


Page 21 of 30


dominantthatitswampsthepolicycomplex,eitherpositivelyornegatively,makingitseffectsnegligible.Forinstance,thereisnopointofferingalowcholesteroldiettoimprovelongevitytoamanwhowillbeexecutedinthemorning.NorininstallingafancyelectroniclockonNancysoldRoversedansince,herdaughterassuresher,thereisnochancethatitwillbestolen.

Soifwearecontenttosettlefortheclaimthatthepolicywillmakeanimprovementonwhatwouldotherwisehavebeenthecasewerethepolicynotimplementedandwehavegoodenoughreasontothinkthatnothingwillswamptheeffectsofthepolicy,thenwearejustifiedinfocusingjustonthepolicyvariableandthefactorsnecessaryforittosucceedinproducingthetargetedeffect.

Awarningreminderisworthmakinghowever.Weallknowthatasuccessfulpolicyonethatdidindeedproduceanimprovementoverwhatwouldhavebeencaneasilybejudgedafailureifitdoesnotproduceanimprovementoverwhatusedtobe.Policyconsumersareapttobeunimpressedbythe(p.316) claim:Yesthingshavegotworse.Buttheywouldhavebeenfarworsestillifwehadntactedaswedidevenifitistrue.Inthesecasesoneneedstohaveagoodaccountofwhatothercausesoperatedtocounterthepolicyeffectsandgoodevidencethatthatisreallythecorrectstory.

V.Mechanisms:aprincipleinaidofpracticaladviceTheprimarypurposeofthetheoryofevidenceforuseistoprovideprincipledgroundsforpracticaladvice.Tothisendweproposetoborrowonemoretenetfromourcolleaguesinphilosophytoaddtothebasicprinciplesofthetheory,albeitonemoreinformallyput.

Principle3:Mechanismsmatter.MethodologistslikeRCTsinpartbecauseRCTsprovideevidenceforcausalrelationswithoutourhavingtoknowthemechanismsbywhichthecauseproducesitseffect.Policymakersgenerallysharethislackofinterestinmechanisms.Theyareconcernedonlywithwhetherthepolicywillproducethetargetedresultsanddonotcareaboutthemechanismsthatwilldrivetheresult.Still,whenwewanttotrytoputacausetowork,gettingabetterunderstandingofthemechanismcanmakeabigdifference.Theimportanceofmechanismsforcausaldiscovery,causalunderstanding,andcausalpredictionhasbeenheavilystressedinrecentphilosophicalliterature.Whatthoughisamechanism?

Causationisalltherageinphilosophynow;mechanismsarecentrestageinthediscussion.Notsurprisinglythenthereareawidevarietyofdifferentcharacterisationsonoffer.21Herewearenotgoingtorelyonanyofthese(p.317) (includingCartwrights)sincetheyaregenerallybothtoonarrowandtooabstracttobeofhelptothosenon-expertinthesciences.Rather,wemakeuseofaninformalnotionofmechanismcommontomanyoftheformalaccounts.Thisisanotionthatcanprovideahelpforpolicymakersaprodfortheimaginationinidentifyingtheauxiliaryfactors(theotherINUSconditions)thatarenecessaryalongwiththepolicyvariabletoproducethetargetedeffect.Forthesepurposeswetakeamechanismtobeananswertothequestion:


Page 22 of 30


Howwouldthepolicyvariablebringaboutthedesiredeffect?Twodifferentwaysofansweringcanhelpinfindingauxiliaryfactors:

1Traceoutthecausalpathwayfrompolicyvariabletoeffect.Seeingwhatshouldcomenextateachstephelpsfocusonwhatwouldberequiredinadditiontothepolicyvariabletomakethenextstephappen.2Manysocialresultsareachievedbycallingintoplaygeneral,oftenfamiliar,routinephenomena,suchasloyalty,mother-love,fearofpunishment,desiretoconform,desiretoberecognised.Differenthelpingfactorswillberequiredbesidesthepolicyvariabletosetdifferentgeneralmechanismsintooperation.Sorecognisingwhichgeneralmechanismswillbecalledoncanbeabighelpinidentifyingthenecessaryauxiliaries.

Tracingthecausalpathway:anexamplefromeconomicsRobertLucasfamouslyarguesthatitisgenerallycounterproductiveforgovernmentstointervenetoregulatetheeconomyonthebasisofobservedregularities(1976).Thatsbecausepeoplewillfigureoutwhatishappeningandactdifferently,therebyunderminingtheveryregularitythegovernmentdependsonforpredictingtheeffectsofitspolicies.OneofhisstrikingexamplesisthatofthePhillipscurve,theempiricallyobservedtrade-offbetweeninflationandunemploymentthatwasusedbypolicymakersinthe1950sand1960stocontrolunemploymentviainflation.LucasusesarationalexpectationsmodeltoshowthatthePhilipscurvewillbreakdownifpeopleknowwhatthegovernmentisdoing.Hismodelreflectsastorythatanswersthequestion,Howdoesrisinginflationproducealoweredrateofunemployment?Insodoingitunearthssomecrucialauxiliaryfactorsthathavetobeinplacebesidesinflationifinflationistoreduceunemployment.

WehaveseenaversionofthePhillipscurvealready,above,p.299,inHadweworldenoughandtime.

Accordingtothisequationanincreaseinpshouldmakeforanincreaseinoutput.Wecansupposethatanincreaseinoutputwillinturnleadtoan(p.318) increaseinemployment.Hencetheequationdescribesatrade-offbetweeninflationandunemployment.Butitisofnouseforpolicy,saysLucas.Hisstorygoeslikethis:Howmuchoutputsuppliersproducedependsonthepricetheyexpecttheirgoodtosellforandonwhattheyexpecttheirexpensestobe.IntheLucasmodeltheaveragepriceforgoodsintheeconomyservesasaproxyforexpense.Sointhemodeltheamountofagoodsuppliedinagivenperioddependsontheratioofthepriceofthegoodtotheexpectedeconomy-widepriceforgoodsinthatperiod.Lucasassumesthatsupplierswillbegoodguessersabouttheeconomy-wideprice:Theeconomy-widepricethattheyexpectistheaverageeconomy-widepricethatactuallyobtains.Inthiscaseoveralloutputofagoodwillbeproportionaltotheratioofthepriceofthegoodtothemeanofeconomy-wideprices.Sotheoutputofagoodwillbegreaterwhenthepriceofthegoodexceedsthemeanofpricesacrosstheeconomy.Thatmeansthattherewillbeapositive

(*) = [ ] + y + .yt pt pt1 pt t


Page 23 of 30


relationshipbetweenoutputandpriceincrease.AnothercausalprocessthatwewontdescribeprovidesOkunslaw,underwhichincreasesinoutputleadtoincreasesinemployment.Thetwoprocessestogetherthusimplythatrisingpriceswillreduceunemployment.

Whathappensifthegovernmentdecidestointervenetoincreaseinflationoverwhatitwouldhavebeen?AssumingthatthePhillipscurve(alongwithOkunslaw)stillholds,unemploymentshouldgodown.Notso,Lucasargues,becausesuppliersaregoodestimatorsoftheeffectsofthegovernmentactiononaverageprice.Iftheyknowaboutthegovernmentsactions,theywillpredicttheaveragepricerisethatwillinfactoccur.Theexpressionforoutputofagoodhaspriceforthegoodinthenumeratorand,assumingsuppliersaregoodestimators,averagepriceriseinthedenominator,recall.Sotheriseinpricesuppliersseefortheirproduct,whichappearsinthenumerator,willpromptanincreaseinoutputonlyifitisnotoffsetbytheincreaseintheaveragepricesinthedenominatorthatinflationwillentail.Indeed,ifthedenominatorgoesupproportionatelyfasterthanthenumerator,thegovernmentpolicytoincreasepricesintheeconomycanevencreateadropinoutputandtherebycauseanincreaseinunemployment.

Whereinequation*doweseethisimportantfactortheaverageofeconomy-wideprices?Itishiddenin.Butrehearsingthecausalprocessstep-by-step,asintheLucasstory,bringsitoutofhiding.Theonlywaythatinflationcanincreaseoutputisiftheaveragepricerisethisinvolvesdoesnotresultinanincreaseintheoverallpriceriseexpectedbysuppliersbigenoughtooffsettheriseinpricethesuppliersseefortheirownproducts.Thetrade-offbetweeninflationandunemploymentholdswhenitdoesbecausesuppliersdonotexpecttheoverallriseinprices.ThustherequisitehelpingfactorontheLucasstorytheINUSfactornecessarytoallowinflationtoworkitslowering(p.319) effectsonunemploymentisthefailureofthesupplierstoforetelltheinflation.Thatsuggeststhatifthegovernmentisgoingtosucceedinthestrategyofencouraginginflationinordertoreduceunemploymentithadbetternotletpeopleknowthatthatiswhatitisdoing.

Thiscaseillustratestwopointsofinteresthere.Equationsarenicebecausetheyexpressprecisequantitativerelationships.Still,trueequationsmayleavealotoutandespeciallyalotweneedtoknowforpolicysuccess.Evenequationsthatare100%descriptivelyaccuratecanfailtolayoutalltheINUSfactorsnecessarytoenablethecausetheypicturetoproducetheexpectedeffect.Second,thinkingthroughthecausalprocessstep-by-stepansweringahowquestioncanmakethesehelpingfactorsapparent.

Identifyingthemeansofproduction:acriminologyexampleWequoteanexamplefromNickTilley(2009)atlengthtoillustratehowthinkingaboutthegeneralmechanismscalledintoplaybythepolicyvariableinordertoproducetheeffectcanalsohelpinidentifyingauxiliaryfactors:

Takepropertymarking.Whatisitaboutitthatisexpectedtoworkasacrimepreventionmeasure?Propertymarkingmightincreasetherisktooffendersby


Page 24 of 30


makingitmorelikelythattheywillbecaughtwithstolenproperty,successfullyprosecutedandpunished.Thisinturnmaymean:

1Moreoffendersareincapacitated,2Someoffendersaredeterredfromfuturecrime,3And/orotherprospectiveoffendersaredeterredastheycometoappreciatewhatwillhappentothemiftheytrytocommitthecrime.

Alternatively(orinaddition),theperceivedincreasedriskofapprehension,regardlessofthereality:

4Maylead(some)prospectiveoffendersnottocommitcrimeinthefirstplace.

Forpropertymarkingtoworkinrelationtoanyindividualoffenderinthefirstway,

(a)Propertythatisliabletobestolenhastobemarked,(b)Offendershavetofailtoremoveordisguisethemarks,(c)Authoritieshavetocheckthatpropertythatmightbestolenhaspropertymarksonit,(d)Policehavetolinkthemarkedpropertybacktothosefromwhomithasbeentaken,(e)Thosefoundwiththestolenpropertyhavetobeunabletocookupaplausibleenoughstoryaboutwhytheylegitimatelyhaveitintheirpossession,(f)Theprosecutorhastobepersuadedthatthecaseisworthtakingtocourt,(g)Thejudge/juryhavetobepersuadedbytheevidence,(p.320) (h)Acustodialsentencehastobepassed,and(i)Therehavetobeoffencesthattheincarceratedpersonwouldotherwisebecommittingbutforthefactthatheorsheisinprison.

Forpropertymarkingtoworkinthesecondway,(ai)havetobeinplace,and

(j)thepenaltyhastobesufficientlysalientthattheoffendermakesdecisionsthatdonotleadtofurtheroffencesorwhichleadtofeweroffences.

Forpropertymarkingtoworkinthethirdway(aj)havetobeinplace,and

(k)Prospectiveoffendersneedtoknow,appreciateandsufficientlyfearthepenaltiesappliedthattheywillmakedecisionsnottocommitoffencesthattheywouldotherwisecommit.

Forpropertymarkingtoworkinthefourthway(ak)neednotbeinplace,but,

(l)Prospectiveoffendersmustknowthatpropertyis(ormayverylikely)be


Page 25 of 30


marked(m)Prospectiveoffendersmustbepersuadedthatthemarkingsignificantlyincreasestheirrisksofbeingcaughtandpenalisediftheystealthemarkedgoods,and(n)Theexpectedpenaltiesmustbesufficienttoleadthemtodecidenottocommittheoffencestheywouldotherwisecommit.

Thus,whatmightworktobringaboutacrimedropthroughpropertymarkingdependsoncontextualcontingencies.

TilleyscontextualcontingenciesarejusttheauxiliaryfactorswehavebeentalkingaboutindiscussingINUSconditions,factorsthatmustbeinplacealongwithpropertymarkinginorderforpropertymarkingtobringaboutadropincrime.Focusing,asherecommends,onhowpropertymarkingissupposedtoachievetheseresultsdirectsattentiontotheseessentialfactors.

VI.InsumOuraimhasbeentolaythefoundationsforconstructingacomprehensiveadviceguideforevaluatingpolicyeffectivenessclaims,aguidethatispracticableandatthesametimerestsonsoundgeneralprinciples.Tothisendweproposethreeprinciples.First,policyeffectivenessclaimsarereallycausalcounterfactualsandtheproperevaluationofacausalcounterfactualrequiresacausalmodelthat(i)laysoutthecausesthatwilloperateand(ii)tellswhattheyproduceincombination.Second,causesareINUSconditions,soitisimportanttoreviewboththedifferentcausalcomplexesthatwillaffecttheresult(thedifferentpies)andthedifferentcomponents(slices)thatarenecessarytoacttogetherwithineachcomplex(orpie)ifthetargetedresultistobeachieved.Third,agoodanswertothequestionHowwillthepolicy(p.321) variableproducetheeffectcanhelpelicitthesetofauxiliaryfactorsthatmustbeinplacealongwiththepolicyvariableifthepolicyvariableistooperatesuccessfully.

Aguidebasedontheseprincipleswillhavetohelpusersconstructtheirowncausalmodelsanduseevidencetojudgehowgoodtheyare.Itshouldalsoprovideshortcuts,whatGerdGigerenzerhascalledcheapheuristics,thatcanachievenearenoughthesameconclusionswithlessinput(Gigerenzer,Todd,andtheABCResearchGroup,2000).Mostofthesewillapplyonlyinspecialconditions.Partofthejobbeforeofferingthemtouserswillbetoshowthattheseshortcutsareindeedgoodonesintherightcircumstances,thentodescribethecircumstancesfortheusersinawaythatcanbeunderstoodandapplied.

Allthisissomethingofatallorderforusers.Thatjustmakesourjobhard.Weneedtodothebestwecantohelpthosewhoneedtoevaluateeffectivenessdosoaswellaspossible,eveniftheprocesswillinevitablybeflawed.Recognisingthatitwillbeflawedmeansmakingclearthatpolicyeffectivenessjudgementswillalmostneverbeverysecure;andsofaraspossible,oneshouldhedgeonesbetsonthem.Itdoesnotmeangivingupontheattempttoconstructacausalmodel,oralternativelydefendingthataparticularshortcutwilldoalmostaswell.For,aswehavestressed,whenonebetsonan


Page 26 of 30


effectivenesscounterfactual,oneisbetting,willy-nilly,onthecausalmodelthatunderwritesit.Thewholepointofevidence-basedpolicyisthatbetslikethisshouldbetakenconsciouslyandbeaswellinformedbyevidenceasispracticable.Itsnogoodduckingtheproblem.Wedbetterjustgetonwithfiguringouthowtomakethisassimpleanduserfriendlyaspossible.

References

Bibliographyreferences:

Bechtel,W.andAbrahamsen,A.(2005),Explanation:amechanisticalternative,StudiesinHistoryandPhilosophyoftheBiologicalandBiomedicalSciences,36:42141.

Bohrnstedt,G.W.andStecher,B.M.(eds.)(2002),WhatwehavelearnedaboutclasssizereductioninCalifornia(Sacramento,CA,CaliforniaDepartmentofEducation).

Cartwright,N.(1999),TheDappledWorld(Cambridge,CambridgeUniversityPress).

Cartwright,N.(2007),HuntingCausesandUsingThem(Cambridge,CambridgeUniversityPress).

Cartwright,N.(2009),Causallaws,policypredictionsandtheneedforgenuinepowers,inTobyHandfield(ed.),DispositionsandCauses(Oxford,OxfordUniversityPress),pp.12758.

Cooper,H.,Robinson,J.C.andPatall,E.A.(2006),Doeshomeworkimproveacademicachievement?Asynthesisofresearch19872003,ReviewofEducationalResearch,76:162.

Dunn,W.(2003),PublicPolicyAnalysis(NewYork,PrenticeHall).

Gigerenzer,G.,Todd,P.M.andABCResearchGroup(2000),SimpleHeuristicsthatMakeUsSmart(NewYork,OxfordUniversityPress).

Kolata,G.(2008),NewArenaforTestingofDrugs:RealWorld,NewYorkTimes,4Nov.2008.Accessibleat:.

Lane,D.C.,Monefeldt,C.andRosenhead,J.V.(2000),Lookinginthewrongplaceforhealthcareimprovements:asystemdynamicsstudyofanaccidentandemergencydepartment,JournaloftheOperationalResearchSociety,5:51831.

Lindblom,C.(1979),UsableKnowledge(NewHaven,CT,YaleUniversityPress).

Lucas,R.(1976),EconometricPolicyEvaluation:ACritique,Carnegie-RochesterConferenceSeriesonPublicPolicy,1:1946.

Machamer,P.,Darden,L.andCraver,C.(2000),Thinkingaboutmechanisms,PhilosophyofScience,67:125.


Page 27 of 30


Mackie,J.L.(1965),Causesandconditions,AmericanPhilosophicalQuarterly,2:24564.

Pearl,J.(1995),CausalDiagramsforEmpiricalResearch,Biometrika,82:66988.

Pearl,J.(2000),Causality:models,reasoningandinference(Cambridge,CambridgeUniversityPress).

Reiss,J.(2007),ErrorinEconomics:towardsamoreevidence-basedmethodology(London,Routledge).

Robinson,D.L.(2006),Noclearevidencefromcountriesthathaveenforcedthewearingofhelmets,BritishMedicalJournal,332:7225;numerouslettersinresponse,availableonlineat:.

Tilley,N.(2009)Whatsthewhatinwhatworks??Health,policingandcrimeprevention,inJ.KnutssonandN.Tilley(eds.),EvaluatingCrimeReduction.CrimePreventionStudies,vol.24(Monsey,NY,CriminalJusticePress).

Notes:ProceedingsoftheBritishAcademy,171,291322.TheBritishAcademy2011.

(1)Forinstancewemaywantaneducationalprogramthatmakeschildrenbetteradaptedtolivefull,independentlivesandtobecomecontributingcitizensbutproperscientificmethodrequiresthestudyofpreciselydefined,measurableoutcomes,likereadingscoresonanIowaTestofBasicSkills.

(2)ForexamplesofthemanyothertypesofissuesthatneedconsiderationseesectiononEffectivenessatpp.294below.

(3)Ofcoursetherewillseldombeahighlycertainyesornoanswer.Soatsomepointanassessmentoftheprobabilitieswillhavetobemadeinlightoftheevidence,evenifonlyroughly.Butreasonableprobabilityassessmentsdependfirstonunderstandingthestructureoftheproblem,whichisthetopictobetackledfirst.

(4)But,asmentionedaboveinfn.1,therearemanyimportantaspectsofthisissuethatwewillnotdiscusshere,includinghowtorelatetheconceptsofscientificstudiestothoseinwhichgoalsareoftenframed.

(5)Isrelevancereally,aswesay,aquestionfortheuserratherthantheknowledgeproducer?Manythinknot.Indeeditisacommoncriticismofstudiesinthesocialsciencesthattheydonotsaywhattheyshow,whattheresultsbearon,atapracticallevel.Wedontthinktheycan.Perhapstheycandobetter,buttherewillalwaysbeagreatnumberofrelevancejudgementsthatmustbelefttotheuser.Whetheragivenfactisrelevantasevidenceforagivenclaimdependsonahostofotherassumptions,boththeoreticalandlocaltothesituation.(ThisisthelessonofthefamousDuhemQuineprobleminthephilosophyofscience.)Forcausalcounterfactualsofthekindweassessin


Page 28 of 30


effectivenessevaluations,relevancewilldependinadditiononhowthecauseissupposedtoproducetheeffect.(SeePartVbelow.)

(6)Thesearecommonlycalledcounterfactualsdespitethefactthatitisgenerallypossiblefortheantecedenttoobtain,andwereittoobtain,theconsequentwouldobtainaswell,ifthecounter-factualistrue.Somefindsubjunctiveconditionalamoreaptlabel,butthetermcounterfactualiswhatisgenerallyusedthroughoutphilosophyandwewillfollowthatusagehere.

(7)ArefereeexpressesconcernovertheconceptsNaturescausesandNaturescalculations.PerhapsthisisanexpressionofaDavidHume-inspiredscepticismaboutcauses.Thereis,however,alarge,articulateandcompellingbodyofliteraturearguingthatcontrarytothisscepticalpositioncausalnotionsmakegoodsenseandareessentialforausefulandaccuratedescriptionofthenaturalandsocialworldandespeciallyforunderstandingandevaluatingclaimsabouttheeffectsofintervening.

(8)Thankstoarefereeforencouragingustomentionthis.TherefereealsosuggestsconsultingWilliamDunn,2003andCharlesLindblom,1979.

(9)Remembering,asarefereestresses,toincluderecipientreactionsthatcanaffecttheoutcome.Forinstanceastherefereepointsout,Whethersomethingiseffectiveinapublicpolicysystemdependsonwhetherpeoplelikethepolicyoutcome,oreventhepolicymechanisminitsownright(e.g.inthecaseofsomeeffectiveorcoercivelabourmarketandwelfarepolicies).

(10)Thisinformationplusthegraph,assumingthegraphiscausallycorrectandtheBayes-netsaxiomsaresatisfied,istantamounttohavingthefullprobabilitymeasureoverallthevariablesinthegraph.Itisthuspossibletopredicttheprobabilityofanyoutcomeconditionalonvaluesofantecedentvariables,whichnaturallycanbeveryuseful.Butthisraisesanimportantpointaboutmodellingtopredictsingularcounterfactuals.Afullprobabilityovertherelevantvariableswillallowustopredicthowprobableadesiredeffectisgiventhatthepolicyvariablestaketheproposedvalues.Butonlyiftheprobabilityisovertheindividualeventsthatwillbeimplementedinthespecificwaytheywillbeatthespecificplaceandtimeunderconsideration.Itisjustthisprobabilitythatissodifficulttofindifitexistsatall,whichmanyofusdoubt.

(11)Forinstance,oneaxiomrequiresthatimmediatelypriorcausesonthegraphandtheireffectsarealwaysprobabilisticallydependent,whichmeansthatnocausesactbothpositivelyandnegativelybydifferentpathsthatcanceleachother.Asecondrequiresthatafullsetofpriorcausesscreensoffafactorfromanythingexceptitscausaldescendants.Thisimplies,amongotherthings,thatnocausesproducetheireffectsprobabilisticallyintandem.Forinstance,nopurelyprobabilisticcausesproduceaparticulareffectjustincasetheyproduceaparticularsideeffect.Ratheralleffectsareproducedindependentlyofallothers(cf.Pearl,2000).

(12)ManyofthosedevelopingthetheoryofcausalBayesnetsdescribethemasa


Page 29 of 30


methodforcausaldiscovery.Wethinkthatsright.Theyaretoolsontheknowledgeproductionside;awaytosidesteptheneedforRCTsbyestablishingefficacywiththesamedegreeofrigourasanRCTbutusingpopulation,notexperimental,data.TheymayevenbeofmoreimmediaterelevancetopolicythananRCTifthedatacomesfromapopulationreasonablydeemedsimilarintherightrespectstothetargetpopulation.Still,withoutfurtheradditions,theyarenotenoughtoevaluatecausalcounterfactuals.(ThoughseeJudeaPearls(2000)beautifulworkonhowtousethemtoevaluatetheprobabilityofcausualcounterfactuals,giveninputprobabilitiesforexogenousfactorsandgiventhatthespecialBayes-netsaxiomsholdinthesystemunderstudy.)

(13)Hence thasaprobabilitymeasureoverit.Thisvariabledoesnotrefertoanyknownquantitybutservesatoneandthesametimetostandforomittedcausesandmeasurementerrorsandasarepresentationaldevicetoallowadeterministic-lookingequationtorepresentapurelyprobabilisticconnectionbetweenthedesignatedcausesandthedesignatedeffect.

(14)When tisincluded,thecauseswillnotfixavaluefortheeffectbutmerelyitsprobability.

(15)Thatis,allcausesareINUSconditions.ButnotallINUSconditionsarecauses.

(16)ThisnaturallysuggeststhatafeedbackmodelaswiththeA&Estudyabovewouldbeagoodonetotryifonewantstolayoutthestepsinthecausalprocessinaidofproducingwhatiscalledacausalmodelhere.

(17)Assumingtherearenootherforcesatworkandignoringthegenerallynegligiblegravitationalattractionbetweenthetwoobjectsthemselves.

(18)Sometimesweareonlyinterestedinestimatingwhatdifferencethepolicywillmakeandeventhensometimesonlythedirectionofchangesothatwecangetbywithoutanestimateofsize.Forthatweclearlyneedsomewhatlessinformation.TobediscussedinPartIV.

(19)Itisbecauseweareconcernedwithevidenceusersratherthanevidenceproducersthatwedonottalkofexternalvalidity.Externalvaliditystartswitharesultandaskswhereoutsidetheexperimentalcontextitwillobtain.Theanswerisgenerallynotmanyplaces,especiallyforRCTresultswherethereisgoodreasontoexpectthesameresultonlyinsituationswheretheeffecthasthesamesetofcausalfactorsandtheprobabilitiesoverthesearethesame.Theproblemforusersisnothowtousesomespecialnuggetofwell-establishedresultbutratherhowtoassembleandtreatalltheevidencethatcanhelpwithalltheissuesinvolvedinestimatingwhatwillhappenintheirspecificcase.

(20)Complexrelationsbetweenthesufficientcausesarepossiblehowever,sosometimesevenforthesekindsofcasesitisnotagoodideatoignoreothercausalcomplexes.Suppose,forexample,thatadjustingonecomponentcauseofacluster(onesliceofapie)


Page 30 of 30


modifiesanothercomponentcauseofthesameclustertheexampleaboutbicyclehelmetsillustratedthisthen,ifthesecondarymodifiedcomponentisalsoacomponentofanothercluster,theeffectofthesecondsufficientclusterwillbemodified.

(21)Weshalldescribesomeoftheseapproachesinordertostressbycontrastthatnoneofthesearewhatwemeanbymechanismhere.HerewemeanananswertoahowquestionthatcanhelpinfindingINUSauxiliaries.Astoothersensesofmechanism:JudeaPearlexplorescausalmodelsthattaketheformoflinearequations,oneequationforeacheffectvariableontheleft-hand-side,layingoutacompletesetofcausesforitontheright-hand-side.Manypeoplecalltheseequationsmechanisms,asinasimplesupplyanddemandmodelineconomicswheretheequationforthequantitysuppliedissaidtodescribethesupplymechanism;thatforthequantitydemanded,thedemandmechanism.NancyCartwright(cf.Cartwright,1999and2007)talksaboutamechanism(oranomologicalmachine)asafixed(enough)arrangementofpartsthatwhensetrunningcangiverisetostablein-put/out-putrelations.ForourUCSDcolleagueWilliamBechtel,Amechanismisastructureperformingafunctioninvirtueofitscomponentparts,componentoperations,andtheirorganization.Theorchestratedfunctioningofthemechanismisresponsibleforoneormorephenomena(BechtelandAbrahamsen,2005).AlternativelyPeterMachamer,LindleyDardenandCarlCraver(2000)definemechanismsasentitiesandactivitiesorganizedsuchthattheyareproductiveofregularchangesfromstartorset-uptofinishorterminationconditions.

Accessbroughttoyouby: PontificiaUniversidadCatolicadelPeru(PUCP)

Documents

A Theory of Evidence for EvidenceBased Policy