Wigmore charts as methodological heuristic: an outline - EEiM …ebmplus.org/wp-content/uploads/2016/02/Wigmore-heuristic... · 2016-02-19 · Introduction What does a method have

Introduction What does a method have to do? What to do about it? Heuristics Conclusion References

Wigmore charts as methodological heuristic: anoutline

EEiM meeting, UCL

Brendan Clarke - [email protected]

22nd February 2016


Wigmore charts as methodological heuristic: an outline


Problem statement

Explaining Evidence in Medicine project

Aims:

Take evidence ofmechanism seriously

Try to characterisequality of evidence ofmechanisms

Try to integrate evidenceof mechanism andevidence of correlation.

(Clarke et al., 2014)




Problem statement

An objection: no method for working with mechanisms

Patrick Bossuyt




Problem statement

Evidence of mechanism looks complicated/heterogeneous




Problem statement

. . . while evidence of correlations looks simple

http://www.gradeworkinggroup.org/Brendan Clarke - [email protected]



A query about clinical decision making

Clinical decision making

Question: Given 2 possible treatments, how can a cliniciandecide which is best? (A/B decisions)

Usual answer: Evidence from relevant clinical trials





But we need more than trials to make clinical decisions

There are many facets to making an A/B clinical decision beyondevidence from trials. We might think about:

benefits

harms

reference class

price

outcomes

patient preferences, needs and consent

...

We need to know (at least a little) about each of these. But eachare not equally important - e.g. non-consent trumps anything else





Clinical decision making is distributed between actors

benefits (trialists / Cochrane / NICE)

harms (MHRA / NICE)

reference class (emmm, give me a minute. . . )

price (NICE)

outcomes (Clinician and patient)

patient preferences, needs and consent (clinician and patient)

...





So what does this have to do with EBM?

“Evidence-based medicine is the conscientious, explicit andjudicious use of current best evidence in making decisions aboutthe care of individual patients.”

(Sackett et al., 1996)

“The practice of evidence based medicine means integratingindividual clinical expertise with the best available external clinicalevidence from systematic research.”

(Sackett et al., 1996)

If we take these quotes at face value, clinical evidence (e.g. fromtrials) participates in clinical decision-making, but does notconstitute it.





Back to the split context of clinical decision making

Evidence producers and synthesisers deal with benefits,harms, price

Clinicians and patients deal with outcomes, patientpreferences, needs and consent

I don’t know who deals with reference classes (but suspectthat this is often ignored)

This makes many demands on the way that we appraise QE, andrules out lots of plausible ways of working

My pet example: the expert system approach





Given this, methods for appraising the quality of evidence ofmechanism must:

work well across these contexts

be transparent, defeasible, lots of other nice epistemic things

play nicely when introduced to other species of evidence(patient preferences, e.g.)




What to do about it?

Question: how to select and evaluate evidence ofmechanism?




What to do about it?

Contrast: evidence about correlations appears simple

http://www.gradeworkinggroup.org/Brendan Clarke - [email protected]



Analogy: Legal work on evidence

A legal analogy





Transparent, cross-context, simple, et al

Testimony

Material evidence - almost anything you can think of (agun, some cocaine)

Hearsay evidence - out of court statements

Documentary evidence - can be any of the preceding threekinds

How do lawyers select the relevant evidence?Who has the expertise to evaluate all of this evidence?





Substance-blind tools for working with evidence

How do lawyers work with all this evidence?

“One answer is provided if we choose to ignore itssubstance and focus instead on its inferential properties”

(Anderson et al., 2005, 72)





Wigmore charts

Inference networks

Fallible

Edges (connections) areprobability modifiers

Easily showindependent/dependentsources of support

Atomic model ofevidence: the overallinference stands or falls(potentially) on singlebits of evidence

(Anderson et al., 2005, 128)Brendan Clarke - [email protected]




Wigmore chartsExamples

Ultimate probandum: xmurdered y

Penultimate probanda: xhad the means to murder y, xhad the motive to murdery,. . .

Generalizations: y’s murdertook place in y’s house, xneeded to be in y’s house tohave the means to murdery,. . .

Evidence E: x was seenleaving y’s house on the nighty died





Wigmore charts and clinical decisions

Inference networks - so (plausibly) compatible withinferences about clinical matters

Fallible - like clinical decisions

Edges (connections) are probability modifiers - compatiblewith probabilistic models of reasoning used in medicine

Easily show independent/dependent sources of support -useful for working with complex dependencies in clinicaldecision making

Atomic model of evidence: the overall inference stands orfalls (potentially) on single bits of evidence - again, useful forworking with complex dependencies in clinical decision making





Mechanisms and Wigmore charts

Assertion: There’s (some) connection between causalmechanisms and inference






Question: Does that mean that we can use Wigmore charts toselect and evaluate evidence of mechanism?






(biased) Answer: Well, yes, I think it does.

I think that we can use Wigmore (or similar) charts to find outhow much evidence of mechanism we need to RWT-ise a particularcorrelation






Question: . . . wait . . . how?

Wigmore charts as heuristic devices that can show where evidenceof correlation needs mechanistic support, and vice versa




Chow’s account

Chow 2015: the vagueness of ‘heuristic’

The meaning of‘heuristic’ is not selfevident

Written from a cognitivescience perspective

There are different typesof heuristic




Chow’s account

Heuristics are commonly understood as economicalshortcut procedures that may not lead to optimal orcorrect results, but will generally produce outcomes thatare in some sense satisfactory or ‘good enough’.

(Chow, 2015, 977-8)




Chow’s account

”Rather, heuristics process specific kinds of informationin systematic and predictable ways. This accounts forwhy they work so well in certain domains, but failmiserably in others. According to some researchers, suchas Gigerenzer, heuristics are attuned to specific domainsor environments that exhibit stable informationalstructures, and this is what enables a heuristic to besuccessful in that domain or environment.

(Chow, 2015, 1005)




Chow’s account

Since the 1990s, Gigerenzer and his colleagues haveinvestigated how heuristics can produce reasonable orappropriate solutions in different ecological contexts.. . . Ina recent review article written with Wolfgang Gaissmaier,the following definition is proposed:

HG:

A heuristic is a strategy that ignores part of the information, withthe goal of making decisions more quickly, frugally, and/oraccurately than more complex methods.

(Chow, 2015, 982)




Chow’s account

(Chow, 2015, 1000)




Chow’s account

H1:

Heuristics are cognitive procedures that can be expressed as rulesone reasons in accordance with

(Chow, 2015, 1002)




Chow’s account

”By ‘acting in accordance with a rule’, I propose to meanthat a rule is guiding one’s behaviour and that there is acausal link between the rule and the behaviour—inFodor’s terms, the rule is implicated in the etiology of thebehaviour that accords with the rule—but the rule is notan intentional object in cognition. In terms of reasoning,reasoning in accordance with a rule means that the rulehas a causal role in the reasoning process, but is notrepresented. Notice that in this way a rule still(non-prescriptively) regulates, governs, or describes areasoning process.”

(Chow, 2015, 1002)




Chow’s account

H2:

H2: Heuristics are satisficing cognitive procedures that can beexpressed as rules one reasons in accordance with.

(Chow, 2015, 1005)




Chow’s account

Here, we’re using ”satisficing” in Herbert Simon’s sense:

”Simon ([1957]) introduced the notion of satisficing todescribe procedures that do not aim to meet thestandards of rational decision theory (or any theory ofrationality governed by the dictates of logic andprobability theory), but instead aim to meet someaspiration level that is in some sense ‘good enough’relative to the desires, interests, and goals of the agent.”

(Chow, 2015, 1003)




Some conclusions

This is a project of making explicit and structured what iscurrently already (probably, partially) happening.

Being explicit is a tool for improving work with evidence.




Translation work needed!

Point of dis-analogy (legal):

“The field of law is unique in that the law defines thematerial facts or the specific propositions or elementsthat are necessary and sufficient to prove some ultimateprobandum.”

(Anderson et al., 2005, 61)

Point of dis-analogy (cognitive science/Chow)

H2:

H2: Heuristics are satisficing cognitive procedures that can beexpressed as rules one reasons in accordance with.

(Chow, 2015, 1005)Brendan Clarke - [email protected]



Towards a holistic model of clinical decision making - I

EBM point: “Evidence-based medicine is the conscientious,explicit and judicious use of current best evidence in makingdecisions about the care of individual patients.” - extendingthe range of evidence to better achieve this

Wigmore charts are substance-blind. Many kinds of evidencecan be included (about efficacy, or harms, or consent, e.g.).




Towards a holistic model of clinical decision making - II

Wigmore charts are flexible, and can be constructed withdifferent emphases or degrees of detail. From “should x belicenced?” to “should I take x?”. Elegantly accomodate thesechanges:

we might take, e.g. consent as given in the policy contextwe might take inferences about pricing as read inprescriber-patient interactions




Acknowledgements

Many thanks to the AHRC for support under

grant AH/M005917/1

Project website: www.ebmplus.orgBrendan Clarke - [email protected]



References

Anderson, T., Schum, D., and Twining, W., 2005. Analysis of Evidence. Cambridge:Cambridge University Press.

Chow, S.J., 2015. “Many Meanings of ’Heuristic’.” The British Journal for thePhilosophy of Science, 66(4): 977–1016.

Clarke, B., Gillies, D., Illari, P., Russo, F., and Williamson, J., 2014. “Mechanisms andthe Evidence Hierarchy.” Topoi, 33(2): 339–360. URLhttp://dx.doi.org/10.1007/s11245-013-9220-9.

Sackett, D.L., Rosenberg, W.M.C., Gray, J.A.M., Haynes, R.B., and Richardson, W.S.,1996. “Evidence based medicine: what it is and what it isn’t.” BMJ, 312(7023):71–72. URL http://dx.doi.org/10.1136/bmj.312.7023.71.



http://dx.doi.org/10.1007/s11245-013-9220-9

http://dx.doi.org/10.1136/bmj.312.7023.71

Documents

Wigmore charts as methodological heuristic: an outline - EEiM …ebmplus.org/wp-content/uploads/2016/02/Wigmore-heuristic... · 2016-02-19 · Introduction What does a method have