ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION: CONTRIBUTIONS FROM LAW

DAVID A SCHUM, PROFESSOR

SCHOOL OF INFORMATION TECHNOLOGY & ENGINEERING

SCHOOL OF LAW

GEORGE MASON UNVERSITY

JON R. MORRIS, AFFILIATE PROFESSOR

SCHOOL OF INFORMATION TECHNOLOGY & ENGINEERING

GEORGE MASON UNVERSITY

CARDOZO SCHOOL OF LAW

JANUARY 20, 2007

SOME POINTS WE WILL ADDRESS IN OUR TALK

• THE FIELD OF LAW HAS GIVEN ALL OF US A VERY RICH LEGACY OF EXPERIENCE AND SCHOLARSHIP REGARDING THE PROPERTIIES , USES AND DISCOVERY OF EVIDENCE. WE WISH TO APPLY SOME OF THIS LEGACY IN THE VERY DIFFICULT TASK OF ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION.

• SOME CONCERNS ABOUT ATTRIBUTES OF THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES.

• SOME THOUGHTS FROM EPISTEMOLOGY, LAW, AND COMMON EXPERIENCE SUGGEST WHAT COMPETENCE AND CREDIBILITY ATTRIBUTES OUGHT TO BE CONSIDERED.

• MAJOR CONTRIBUTIONS FROM LAW: A LEGACY OF IMPORTANT QUESTIONS TO ASK ABOUT THE COMPETENCE AND CREDIBILITY ATTRIBUTES OF HUMAN WITNESSES.

• PROBABILISTIC ISSUES IN THE ANALYSIS OF CHAINS OF REASONING IN WHICH CREDIBILITY CONSIDERATIONS FORM THE FOUNDATION FOR THESE CHAINS.

• THESE VARIOUS THOUGHTS ARE COMBINED IN A COMPUTER-BASED SYSTEM WE HAVE BEEN WORKING ON FOR QUITE SOME TIME TO ASSIST OUR INTELLIGENCE ANALYSTS DO A MORE CAPABLE JOB OF ASSESSING SOURCES OF HUMINT [HUMAN INTELLIGENCE]. THIS SYSTEM IS CALLED "MACE" [METHOD FOR ASSESSING THE CREDIBILITY OF EVIDENCE]. TWO IMPORTANT USER COMMUNITIES FOR MACE: DEPARTMENT OF DEFENSE AND LAW ENFORCEMENT.

2

LAW AND INTELLIGENCE ANALYSIS

• AMONG THE MOST VALUABLE CONTRIBUTIONS FROM LAW IS THE LEGACY OF EXPERIENCE AND SCHOLARHIP ON WITNESS COMPETENCE AND CREDIBILITY ASSESSMENT ACCUMULATED OVER THE PAST FIVE HUNRED YEARS OR SO IN OUR ANGLO-AMERICAN ADVERSARIAL LEGAL SYSTEM

• WILLIAM TWINING ADVISES THAT WE SHOULD EMPHASIZE EXPERIENCE, SINCE MOST OF THESE INSIGHTS ARISE IN THE DAY TO DAY CRUCIBLE OF ADVERSARIAL EXPERIENCE IN TRIALS AT LAW.

• WE HAVE BEEN ATTEMPTING TO EXPLOIT THIS LEGACY OF WISDOM IN OUR WORK FOR INTELLIGENCE ANALYSTS AND OTHERS WHO PROVIDE THE HUMAN INTELLIGENCE [HUMINT], ON WHICH SO MUCH DEPENDS.

• OUR ANALYSTS ARE ASKED TO PERFORM CREDIBILTY ASSESSMENTS UNDER AN ALMOST UNBELIEVABLE ARRAY OF DIFFICULTIES AND NEED ALL THE HELP THEY CAN GET.

• YOU WILL FIND NO BETTER ACCOUNT IN OPEN SOURCES OF THESE DIFFICULTIES THAN BY READING ABOUT OUR EXPERIENCE WITH "CURVEBALL" IN THE BOOK: ON THE BRINK, BY TYLER DRUMHELLER AND ELAINE MONAGHAN [CARROLL & GRAF, NY, 2006].

• IN THESE PERILOUS TIMES THE IMPORTANCE OF HUMINT CANNOT BE OVERSOLD. WHAT IS SO IMPORTANT ARE OUR EFFORTS TO DETERMINE WHETHER WE CAN BELIEVE WHAT HUMAN SOURCES TELL US.

3

FIRST CONTRIBUTION FROM LAW:

DISTINGUISHING BETWEEN THE COMPETENCE AND THE CREDIBILITY OF HUMAN SOURCES

COMPETENCE:

• Appropriate sources,

• In a position to observe,

• Understanding of what was observed,

• Ability to communicate.CREDIBILITY:

• Veracity,

• Objectivity

• Observational Sensitivity

ORTHOGONAL CHARACTERISTICS:ONE DOES NOT ENTAIL THE OTHER.

4

THESE TWO CHARACTERISTIC ARE FREQUENTLY CONFUSED LEADING TO SERIOUS INFERENTIAL ERRORS.

EXAMPLE OF A FREQUENT MISTAKE:

“We can believe what X tells us because he had good access to his sources”

Arguments jusifying theseattributes in a minute.

A SEARCH FOR THE ATTRIBUTES OF THE CREDIBILITY OF HUMAN SOURCES OF EVIDENCE:

THE “STANDARD ANALYSIS”OF KNOWLEDGE AND THE CHAIN OF REASONING IT SUGGESTS [controversial, but a good heuristic]

5

A SOURCE “KNOWS” THAT EVENT E OCCURRED IF:

E DID OCCUR,

THE SOURCE GOT NON-DEFECTIVE EVIDENCE THAT E OCCURRED, AND

THE SOURCE BELIEVED THE EVIDENCE THAT E OCCURRED.

THIS SOURCE NOW TELLS US THAT E OCCURRED BASED ON AN OBSERVATION HE/SHE ALLEGEDLY MADE. WE HAVE UNCERTAINTY ABOUT THESE THREE EVENTS.

OUR INFERENCE ABOUT EVENT E, BASED ON THIS SOURCE’S TESTIMONY E*

HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR?

DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE?

DOES THIS SOURCE BELIEVE THAT E OCCURRED?

THE SOURCE’S REPORT E* THAT EVENT E OCCURRED.

OBSERVATIONAL SENSITIVITY

OBJECTIVITY

VERACITY

HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR?

DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE?

DOES THIS SOURCE BELIEVE THAT E OCCURRED?

THE SOURCE’S REPORT E* THAT EVENT E OCCURRED.

SOURCES OF SUPPORT FOR THIS CREDIBILITY-RELATED CHAIN OF REASONING:

VERACITY

OBJECTIVITY

OBSERVATIONALSENSITIVITY

WIGMORE

WAS THIS EVIDENCE ADEQUATE?

6

WITNESS TESTIMONY BASED ON PERSONAL OBSERVATION

WITNESS DO NOT ALWAYS TESTIFY WHAT THEY BELIEVE

DID THE WITNESS BASE A BELIEF ON SENSORY EVIDENCE/

COMMON EXPERIENCE

A PERSON TELLS US WHAT HE/SHE OBSERVED

PEOPLE DO NOT ALWAYS BELIEVE THE THINGS THEY TELL US

PEOPLE DO NOT ALWAYS BELIEVE WHAT THEIR SENSES RECORD, BUT BELIEVE WHAT THEY EXPECT OR WISH TO HAVE OCCURRED

HUMAN SENSE ARE FALLIBLE, ESPECIALLY UNDER CERTAIN CONDITIONS

7

A MAJOR CONTRIBUTION FROM LAW: GROUNDS FOR IMPEACHING OR SUPPORTING WITNESS CREDIBILITY

VARIETY IS THE SPICE OF LIFE IN CREDIBILITY ASSESSMENT

DECOMPOSING THE LINK BETWEEN EVIDENCE E* AND EVENT E

LP E H h f f

P E H h f fE*

( | )[ ]

( | )[ ]=

− +− +

1

2

For h = P(E*|E) =

P(ES|E)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB

C)] + P(EB|ESC)

[P(E*|EB) - P(E*|EBC)] + P(E*|EB

C).

For f = P(E*|EC) =

P(ES|EC)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB

C)] + P(EB|ES

C)[ P(E*|EB) - P(E*|EBC)] +P(E*|EB

C).

8

SOURCE”S TESTIMONY E*

DOES THE SOURCE BELIEVE THAT E OCCURRED?

P(E*|EB) AND P(E*|EBC)

[VERACITY]

DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE OF E?

P(EB|ES) AND P(EB|ESC)

[OBJECTIVITY]

DID EVENT E OCCUR? [HOW GOOD WAS THE SENSORY EVIDENCE?]

P(ES|E) AND P(EB|EC)

[SENSITIVITY]

ONE PART OF THE MACE SYSTEM NOW DESCRIBED USES THESE ALGORITHMS FOR DETERMINING h AND f.

THE MACE SYSTEM:

MACE = METHOD FOR ASSESSING THE CREDIBILITY OF EVIDENCE

MACE INITIALLY DESIGNED FOR CIA IN 1990; BUT THEY WERE NOT READY FOR IT. SINCE THE EVENTS OF 9/11/01 AND SUBSEQUENT EVENTS, THEY ARE MORE THAN READY FOR IT NOW. MACE CONSISTS OF TWO PARTS LEADING TO TWO DIFFERENT HEDGES ON A CONCLUSION ABOUT WHETHER TO BELIEVE WHAT A HUMAN SOURCE TELLS US.

PART I.

HOW COMPLETE IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BACONIAN QUESTION]. MACE ALLOWS THE USER TO:

• MARSHAL ANSWERS TO THE COMPETENCE AND CREDIBILITY QUESTIONS SUGGESTED BY 500 YEARS OF EXPERIENCE IN LAW,

• JUDGE WHETHER THE ANSWERS FAVOR OR DISFAVOR THE SOURCE’S COMPETENCE AND ATTRIBUTES OF HIS/HER CREDIBILITY,

• DECIDE WHETHER THE EVIDENCE JUSTIFIES A BELIEF IN WHAT THE SOURCE HAS REPORTED [THIS PART MIGHT STAND ON ITS OWN].

PART II.

HOW STRONG IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BAYESIAN QUESTION]. MACE ALLOWS THE USER TO:

• ASSESS LIKELIHOODS ASSOCIATED WITH THE

THREE CREDIBILITY ATTRIBUTES, BASED ON EVIDENCE MARSHALED IN PART I.

• COMBINES THESE LIKELIHOOD ASSESSMENTS USING THE ALGORITHM JUST GIVEN.

• CALCULATES POSTERIOR ODDS ON WHETHER WE SHOULD BELIEVE WHAT THIS SOURCE HAS TOLD US.

9

PART I IN MACE ALLOWS THE MARSHALING OF ANSWERS TO THESE COMPETENCE AND CREDIBILITY QUESTIONS COMING FROM LAW

COMPETENCE

1. ACCESS TO INFORMATION

2. KNOWLEDGE AND

EXPERTISE.

3. OBSERVATIONAL

CAPABILITIES.

4. MEETING BEHAVIOR

5. MOTIVATIONAL

CONSISTENCY

6. RESPONSIVENESS.

VERACITY

1 PRIOR INCONSISTENCIES

2. OUTSIDE INFLUENCES

3. EXPLOITATION POTENTIAL

4. CONTRADICTION AND

CONFLICT

5. CORROBORATION AND

CONFIRMING

6. CHARACTER

7. REPORTING RECORD

8. TAILORING INFORMATION

9. COLLATERAL DETAILS

10. INTERVIEW BEHAVIOR

OBJECTIVITY

1.OBSERVATIONAL

EXPECTATIONS

2.OBSERVATIONAL DESIRES

3. BELIEF CONSEQUENCES

4. MEMORY EFFECTS

5. CONTRADICTION AND

CONFLICT

SENSITIVITY

1. SENSORY CAPACITY

2. OBSERVATIONAL

CONTEXT

3. PAST ACCURACY

4. CONTRADICTION AND

CONFLICT

5. COLLATERAL DETAILS

USERS RESPOND TO ANSWERS TO THESE QUESTIONS BY CONSIDERING THE EVIDENCE AT HAND FOR THE SOURCE AND CHECKING ONE OF THE FOLLOWING BOXES:

ANSWER IS FAVORABLE TO AN ATTRIBUTE:

ANSWER IS DISFAVORABLE TO AN ATTRIBUTE:

I CAN’T DECIDE:

NO ANSWERS TO THIS QUESTION:

10

OVERALL BACONIAN ASSESSMENT IN PART I OF MACE

YOUR OVERALL JUDGMENT ABOUT THIS REPORT:

INTELLIGENCE SUPPORTS THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR:

INTELLIGENCE FAILS TO SUPPORT THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR:

CAN’T DECIDE:

JUDGEMENT BASED ON THIS SUMMARY:AN EXAMPLE

ANSWERED QUESTIONS UNANSWERED QUESTIONS

COMPETENCE [5 Qs]

VERACITY [10 Qs]:

OBJECTIVITY [5 Qs]:\

SENSITIVITY [5 Qs}:

2 1 2

52 3

CREDIBILITY SUMMARY:

1 1 3

2 2 1

2 3 8 7 11

MACE PART II. BAYESIAN ASSESSMENTS OF THE STRENGTH OF EVIDENCE REGARDING THE CREDIBILITY OF A HUMINT SOURCE.

ASSESSMENTS OF PAIRS OF LIKELIHOODS FOR EACH CREDIBILITY ATTRIBUTE:

1.0

1.00

12

1.0

1.00

FOR VERACITY FOR OBJECTIVITY

1.0

1.00

FOR SENSITIVITY OR ACCURACY

P(E*|EB)

P(E*|EBC)

P(EB|ES)

P(EB|ESC)

P(ES|E)

P(ES|EC)

• ASSESSMENTS BASED ON ALL EVIDENCE FOR EACH ATTRIBUTE

• ASSESSMENTS CAN BE MADE IN GRAPHICAL FORM AS SHOWN ON THE NEXT SLIDE.

AN EXAMPLE OF LIKELIHOOD ASSESSMENT: FOR VERACITY

ANALYSTS, AS WELL AS OTHERS, RESIST HAVING TO MAKE NUMERICAL ASSESSMENTS OF PROBABILITIES. SO, MACE ALLOWS THEM TO DRAW BOXES IN A TWO DIMENSIONAL PROBABLITY SPACE [ONE BOX PER ATTRIBUTE].

13

1.0

1.00

P(E*|EB)

P(E*|EBC)

LOCATION OF BOX TELLS HOW STRONGLY THE EVIDENCE FAVORS OR DISFAVORS VERACITY.

SIZE OF BOX SHOWS USER CONFIDENCE IN HIS/HER ASSESSMENT

AUTOMATIC CALCULATIONS MADE BY THE MACE SYSTEM

FIRST: MACE FINDS THE FOUR NUMERICAL CORNER POINTS OF EACH OF THE THREE BOXES DRAWN BY THE USER; ONE BOX PER ATTRIBUTE.

SECOND: MACE FORMS ALL 64 COMBINATIONS OF THESE CORNER POINTS.

THIRD, FOR EACH COMBINATION, MACE CALCULATES h = P(E*|E) AND f = P(E*|EC) USING THE ALGORITHM PREVIOUSLY GIVEN:

For h = P(E*|E) =

P(ES|E)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB

C)] + P(EB|ES

C)[P(E*|EB) - P(E*|EBC)] + P(E*|EB

C).

For f = P(E*|EC) =

P(ES|EC)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB

C)] + P(EB|ES

C)[ P(E*|EB) - P(E*|EBC)] +P(E*|EB

C).

FORTH, MACE FINDS ALL POSSIBLE VALUES OF THE 64 POSSIBLE h/f = P(E*|E}/P(E*|E) RATIOS AND FINDS AN INTERVAL CONTAINING ALL OF THEM. NOTE:THE RATIO h/f = P(E*|E}/P(E*|E) SHOWS THE BAYESIAN FORCE OF EVIDENCE IN AN INFERENCE ABOUT WHETHER E IS TRUE, BASED ON THE SOURCE’S REPORT E*.

14

FINALLY, MACE PROVIDES THE FOLLOWING GRAPHICAL DISPLAY: A SINGLE EXAMPLE

A LOG ODDS OR LOG LIKELIHOOD RATIO SCALE

0 __

FAVORS EC +FAVORS E

PRIOR ODDS INTERVAL. GIVEN INITIALLY BY USER

LIKELIHOOD RATIO INTERVAL CALCULATED BY MACE

POSTERIOR ODDS ON E , GIVEN THE SOURCE’S REPORT.

15

WHAT'S NEXT IN THE DEVELOPMENT OF MACE?

• EXTENDING MACE TO DEAL WITH ADDITIONAL CREDIBILITY MATTERS WHEN HUMAN SOURCES PROVIDE TANGIBLE EVIDENCE.

• EXTENDING MACE TO COPE WITH “WILDERNESS OF MIRRORS” PROBLEM, WHEN WE HAVE CHAINS OF SOURCES ALL COMMENTING ON EACH OTHER’S CREDIBILITY.

• ENHANCING MACE TO ALLOW BETTER RECOGNITION OF DECEPTIVE HUMAN BEHAVIOR.

MANY THANKS FOR ALLOWING US TO SHARE WITH YOU PARTS OF THE STORY OF MACE.

THAT'S ALL FOR NOW!

16