Human Factors in Forensic Science

William C. Thompson Department of Criminology, Law & Society

University of California, Irvine

OLLI January 12, 2016

Joseph Bell

Paul Kirk August Vollmer

Public Perceptions of Forensic Science

• Highly credible • Unbiased • Extremely

valuable for solving crime

Reasons for Wrongful Convictions

Feb. 2009 NAS Report

Scathing Critique: • Entire disciplines rest on deficient scientific

foundations • Procedures for interpretation lack rigor • Inadequate efforts to avoid error and bias • Analysts routinely testify with unwarranted

certainty

Abandoned Method—

Bullet lead evidence

Scientific Foundations

FBI Abandons Bullet Lead Testing—Sept 1, 2005

Bureau’s decision was “based primarily on the inability of scientists or manufacturers to definitively evaluate the significance of an association between bullets made in the course of a bullet lead examination.” --Dwight Adams, FBI Lab Chief

Scientific Foundations: Fire Investigation

• Fire investigators have for decades given mistaken testimony about arson indicators: – Crazed glass – Melted steel – Multiple V-Shaped burn patterns – Pooling/puddling

• Why did this happen?

Problematic Forensic Science Disciplines

• Bitemarks • Hairs

Pattern Matching Disciplines

NAS Findings (2009)

• Inadequate validation in the pattern matching disciplines (e.g., latent print, toolmark, hair & fiber, bitemarks, question documents) – “With the exception of nuclear DNA analysis…no

forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source” (S-5)

– “The simple reality is that the interpretation of forensic evidence is not always based on scientific studies to determine its validity. This is a serious problem.” (S-6)

NAS Findings (2009)

Potential for Bias • “Forensic science experts are vulnerable to

cognitive and contextual bias…” • “…renders experts vulnerable to making

erroneous identifications” • “These disciplines need to develop rigorous

protocols to guide these subjective interpretations…”

• “…to date there is no good evidence that the forensic science community has made a sufficient effort to address the bias issue…”

Bias—inappropriate influence

Bias in forensic science

• Being influenced inappropriately • by something other than (or in addition to) the

physical evidence designated for examination • such as task-irrelevant contextual information

Laber et al. (2014) NIJ Blood Pattern Analysis

Study

• When witness’ statements mentioned a gunshot, pattern was more likely to be interpreted as “high velocity” splatter

• When witness’ statements mentioned coughing, pattern was more likely to be interpreted as “expiration”

• These effects were stronger when – Less information was available (smaller sample) – Pattern more ambiguous due to background

NAS Report on Forensic Science (2009)

• “Forensic science experts are vulnerable to cognitive and contextual bias…”

• “…renders experts vulnerable to making erroneous identifications”

• “These disciplines need to develop rigorous protocols to guide these subjective interpretations…”

• “…to date there is no good evidence that the forensic science community has made a sufficient effort to address the bias issue…”

Nakhaeizadeh, Dror & Morgan (Science & Justice, 2014)

41 Forensic Anthropologists

Context Suggests % Saying Male Male 72 Female 0 Neither 31

Context Suggests % Saying Caucasian Caucasian 100 Asian 50 Neither 100

Context Suggests

Experts Say 18-25 26-35 36-45 46+

25-30 0 78 22 0 50-55 0 14 50 36

Nothing 8 46 38 8

Studies Showing Contextual Bias in Forensic Science

• Document examination (Miller, 1984) • Fingerprint interpretation (Dror, Charlton &

Peron, 2006; Dror & Rosenthal, 2008) • Crime scene analysis (Helsloot and

Groenendaal, 2011) • Bite mark analysis (Osborne, Woods, Kieser &

Zajac, 2013) • DNA Interpretation (Dror & Hampikian, 2011) • Blood spatter analysis (Taylor et al. 2014) • Forensic Anthropology (Nakhaeizadeh, Dror &

Morgan, 2014)

How contextual bias occurs

Contextual information: • Creates expectations and/or desires • which influence interpretation of evidence • A process known as:

– Context effect – Observer effect – Examiner bias – Confirmation bias

• (although the “bias” may occur without awareness)

Context Influences Perception

Context Influences Perception

Context Influences Perception

• Expectations (arising from context) influence interpretation of visual stimuli

• Expectations/theories can also influence interpretation of more complex types of evidence – Context effects – Observer effects

“…one of the most venerable ideas of traditional epistemology…”

• “The human understanding when it has once adopted an opinion draws all things else to support and agree with it. And although there be a greater number and weight of instances to be found on the other side, yet these it either neglects or despises, or else by some distinction sets aside and rejects, in order that…the authority of its former conclusions may remain inviolate.” – Francis Bacon (Novum Organum, 1620) – See also, Itiel Dror, How can Francis Bacon help forensic science?

50 Jurimetrics 93 (2009)

“…one of the better demonstrated findings of 20th century

psychology…”

• Hundreds of demonstrations • The influence may be:

– Motivational—affecting disposition or motive to reach a particular result

– Cognitive—affecting interpretation and assessment of data

• It affects all human beings • particularly when evaluating ambiguous data • It occurs largely without conscious awareness

Influence without Awareness

• People often think they were influenced by factors that do not affect their judgments

• People’s judgments are often influenced by factors they do not realize are influential

• Reflecting limited awareness of underlying cognitive process

Flexible standards; shifting criteria

Example: Choosing a police chief • What is more important, practical experience

(“street smarts”) or academic record? • Depends on candidate gender

Blinding as a Solution to Contextual Bias: Orchestra Auditions

• Women 5x more likely to be selected in blind auditions

• % of women in top 10 orchestras – 1970s—5% – Today—25%

Blinding is not common in forensic science

From crime lab notes: – “D. Aboto [prosecutor] left msg. stating this S. is

suspected in other rapes but they cant find the V. Need this case to put S away.”

– “Suspect-known crip gang member--keeps ‘skating’ on charges-never serves time. This robbery he gets hit in head with bar stool--left blood trail. Miller [deputy DA] wants to connect this guy to scene w/DNA …”

– “Death penalty case! Need to eliminate Item #57 [name of individual] as a possible contributor”

– “We need you to match [this latent print] to our crook right away because he is about to leave the country”

DNA Testing for Humans and Finches

Ambiguity in DNA Testing?

• Bands in male fraction of vaginal extract match bands of Suspect 2

• But what about Suspect 1?

Ambiguities in DNA Evidence Evidence

Defendant

Contextual Bias?

Does defendant really “match” this evidentiary sample? Defendant D3 vWA FGA Tom 17, 17 15, 17 25, 25 Dick 12, 17 15, 17 20, 25 Harry 14, 17 15, 17 20, 25 Sally 12, 17 15, 15 20, 22

Painting the target around the arrow

Frequency estimates (for Tom): p2 x 2pq x p2

Suspect D3 vWA FGA Tom 17, 17 15, 17 25, 25 Dick 12, 17 15, 17 20, 25 Harry 14, 17 15, 17 20, 25 Sally 12, 17 15, 15 20, 22

See, Thompson, Painting the target around the matching profile. Law, Prob. & Risk (2009)

Federal Regulatory Efforts

• National Commission on Forensic Science – Human Factors Subcommittee

• NIST/DOJ Organization of Scientific Area Committees (OSAC) – Human Factors Committee

NIST/DOJ Organization of Scientific Area Committees (OSAC)

HFC Members

Expertise: • Medical decision making • Airline safety • Military performance • Technology interface • Personnel selection • Readiness for duty • Morale/Motivation • Reduction of bias and error

Role of the OSAC Human Factors

Committee

• Understanding and improving human performance – particularly human judgment and decision making

• Improving efficiency, accuracy, morale of organizations

• Reducing potential for error, bias, misconduct

Reducing Contextual Bias in Forensic Science

• Context Management Systems – Blinding – Delayed Exposure (Sequential unmasking) – Evidence Line-Ups – Independent blind reviews

Objections to Context Management

We need to know EVERYTHING! • “Context is always relevant in forensic science” • “We need to know as much as possible about a

case to do our jobs” • “I won’t allow myself to be influenced by

extraneous information unless it is really credible, and if it is credible what’s that problem?”

• “Is it really a bias if it leads us toward the truth?” “Bias is Good!”

Should forensic scientists be completely blind to

context?

• NO! • Contextual information will often be necessary to

understand and interpret physical evidence – E.g., substrate and method of lifting latent print

Should Conclusions of Forensic Odontologist be Influenced by DNA Evidence?

[After examining a bite mark] …[i]f I then found that DNA [evidence] came back as not excluding that same person, my confidence level would increase. I might be willing to upgrade my opinion from cannot exclude to probable….Now, many odontologists say you shouldn’t have any awareness of the DNA results compared to the bite mark…but if I subsequently get them, then I reserve the right to write a revised opinion. And I have done that.

From a transcript (2012).

How is the DNA relevant?

The DNA matches the defendant

Defendant is the perpetrator

Defendant is probably the source of the bitemark

Key Priniciples

Forensic scientists should draw conclusions: – From the physical evidence designated for

examination • Not from other evidence in the case, • unless that other evidence helps them interpret

the physical evidence – Using valid methods from their discipline that

they are trained to use

Proposed Views/Guidance Statement

Forensic scientists should rely solely on task-relevant information when performing forensic analyses. Information is task-relevant if it is necessary for drawing conclusions:

(i) about the propositions in question, (ii) from the physical evidence that has been designated for examination, (iii) through the correct application of an accepted analytic method by a qualified analyst.

Recommendation

• Forensic scientists in each discipline should: – Think carefully about what types of information

are task-relevant and task-irrelevant for common forensic tasks

– Think about ways to shield analysts from exposure to task-irrelevant information

• Will require modifications in work-flow and separation of responsibilities

The Temptation of the Irrelevant

• Analysts will be tempted to use task-irrelevant information

• Their motive: – not partisanship, venality or malice – truly powerful (especially for expert witnesses):

•The desire to be right

The Criminalist’s Paradox

• By considering “task-irrelevant” information (in an effort to reach the correct conclusion)

• Analysts undermine the probative value of their conclusions for the jury

• By trying to be “right,” analysts increase chances the justice system will go wrong. – See Thompson, Aust. J. Forensic Sci 43(2-

3):123-34 (2011)

Danger of Merging Role of Forensic Scientist and Detective

Being Scientifically Rigorous or Being Sherlock?

• You can do both! • Just not in the same

case.

Special Problems of Medical Examiners

• Iowa v. Tyler (Iowa Supreme Court, June 30, 2015) – Medical examiner should not have been allowed to

express an opinion on the manner of death that was based on non-medical evidence

• But ME can rely on non-medical evidence in reaching conclusion for death certificate

• Dilemma might be resolved through a blind verification process that focuses on what conclusions, if any, can be drawn from medical evidence alone.

Thank you Email: william.thompson@uci.edu