The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports about Past Events

David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman,K. Anders Ericsson, and Laura B. Hassler

Expert Performance Research Team,Human Performance Laboratory

Learning Systems Institute,The Florida State University, USA

www.lsi.fsu.edu/hpl

Using Conventional Interviews to Provide Insights into Cognition during Performance

Conventional interview - common method of obtaining insights into cognitive processes mediating differentially skilled performance

For example, an interview conducted after a military mission to investigate the nature of a decision making during that mission

Problems with Interviews Key problem with interviews and other self-

report measures - tendency by respondents to interpret, explain, summarize thoughts (Nisbett & Wilson, 1977).

Think Aloud Technique - Ericsson and Simon (1993) A task-concurrent cognitive process-tracing method

– used in domains ranging from chess to medical practice (Ericsson & Lehmann, 1996).

Performers think out aloud their thoughts, as if alone, while performing a task

Perform TA exercise

Think aloud technique Example think aloud: Think aloud while identifying the 4th

letter in the alphabet after the letter P.Reports might look like “There’s the P, then there’s err Q, then R, S, and then T, it’s T”.

Thought to elicit verbal reports that reflects contents of WM during task execution

Reduces interpretations, explanations, and summaries of thoughts

Not the performer’s interpreted account of their thoughts but a more direct reflection of their actual thoughts

Constraints on Applicability of Think Aloud Method Physical constraints (e.g., from rapid respiration)

Environmental constraints (e.g., task is undertaken in water)

Event danger/importance (i.e., unwillingness to verbalize during critical events)

In some domains, neither feasible nor desirable to apply think aloud method

Solution: Immediate Retrospective Report Method After completing a task, the participant

reports immediately on the thoughts experienced during the task

Perform TA and IRR exercise

Immediate retrospective report Example alphabet task IRR:

Experimenter: What was the first thought you remember having while coming up with the answer, and the next, and the next, and so on?

Participant: “I saw a P, and then there was a Q, R, S, and finally a T, it was T”.

Limits on IRR Applicability Limited to those situations in which the

researcher can apply the method immediately following task completion

To return to the example of examining mission decision making, access to military personnel might not be possible for some time after the mission

Solution: Delayed Retrospective Report Our research group has been developing a delayed retrospective report (DRR)

method DRR is identical in essence to the IRR but comprises many more directions for

the participant that support the process of recalling thoughts experienced during the event

Incremental approach

1. participant directed to provide a brief overall description of the event2. a timeline around a critical event established that includes clearly

recalled “event markers”3. participant directed to recall thoughts and actions during each event

marker, staring with first, then second, third, and so on

Crucially, like the IRR method, the DRR method directs the participant to:- only report those thoughts and actions that can be recalled with certainty- avoid interpreting, explaining, or summarizing thoughts or actions

Early Attempts to Use DRR with Police–Recall of Thoughts While Performing Under Stress Study

Step 1: Initial overall description of event obtained and establishment of sub-events and their timeline

I was dispatched to an attempted armed robbery call. The robbery was attempted in a man’s yard. Two suspects actually tried to rob an individual who…was armed. There was a stand off, no shots were fired, and the suspects fled.

I chose a route that I thought would most likely bring me into contact with the suspect vehicle. I…meet the suspect vehicle on the road, turned around on it, initiated my overhead lights. The suspect vehicle yielded. I exited my vehicle with my shotgun, conducted a felony [vehicle] stop [procedure], and took the driver into custody.

Step One: The Timeline

Sub-eventmarkers

Rad

io c

all

Car

sto

pped

Car

see

n

Step 2: Identifying Most Stressful Part of Event The most stressful part of the whole encounter was from the

period that I realized that he was going to yield [his vehicle] to me and the point that I actually exited my vehicle and had my shotgun trained on him

I was at my most vulnerable from the time that he stopped and I stopped behind him because the suspect had an advantage then in that he’s already made up his mind what he’s gonna do and if he decides to exit his vehicle quickly with a handgun, he will probably be able to get shots off at me before I can either respond or extract myself from the situation

Delayed Retrospection of Thoughts and Actions using TimelineSection of transcript:

I: So what was the first thought you remember having as, as the vehicle started to yield?

PO: There’s only one guy in the car. I: What was your next thought after that?.... PO: I do absolutely recall thinking “be careful, don’t let the [shotgun]

sling get tangled up on say the gear shift or the radio”…and if it does get tangled up…I need to just drop it and immediately draw my handgun and engage him with that.

I: What was the next thought you remember? PO: I’m, I’m thinking light him up [with the shotgun flashlight] and then

go through a felony stop ritual.

Attempts to Validate DRR Current research in our laboratory with police

officers involves studying the relative validity and reliability of the methods described here

Some Context - http://www.wnbc.com/news/4706691/detail.html

July 11 2005

LOS ANGELES – Police shot and killed a man Sunday night when he emerged from his home carrying a young girl, about 17 months old, following an hours-long standoff and opened fire, wounding one officer. The girl was also killed in the gunfire.

[LAPD Chief McDonnell] said. "The officers are taking it very hard," McDonnell said. "Anytime you have a baby killed, it takes its toll."

Police Simulation in Human Performance Laboratory

Expert Police Officer “Truman”

DRR Validation – Police Study

12 expert (SWAT, TAC) and 12 novice (cadet) police officers (but I’m not going to report here on expert-novice differences)

Initial training in think aloud and IRR techniques with established exercises

In a simulator, officers equipped with modified gun, exposed to 20 film clips depicting representative and stressful police scenarios, requiring shoot/no shoot decision

Video captures behavior of officer – handles gun, unholsters gun, points and aims gun relative to events occurring on film – all verbal and non-verbal commands

For each shot, computer captures initial trigger squeeze time point, full trigger depression time point, and gun shot placement relative to events occurring on screen

IRR captures thoughts experienced during scenario

Police Scenario Simulation,Human Performance Laboratory Plan of equipment

Projector

Officerstandshere

Video record of trigger behavior

Video of whole body behavior

Mirror

ProcedurePresentation # Clip # IRR performed?

1 9 IRR

2 17 No IRR

3 12* IRR

- through to

18 4 No IRR

19 6* IRR

20 2 IRR

DRR is then performed about thoughts and actions undertaken during two(a priori determined) stressful scenarios, indicated with*

Procedure – triangular comparisons Compare reports from IRR obtained immediately after certain

scenarios with events in the film and recorded response behaviors of officer during those scenarios

Compare IRR reports with those obtained from DRR undertaken some time after all scenarios are completed

Compare DRR reports with events in the film and recorded response behaviors of officer during certain scenarios

Towards the “Damn Konfident” In terms of the reliability of

knowledge continuum proposed by Rainer Martens, the DRR method, when compared to a conventional interview method, should help us move away from feeling like we “Don’t Know” and towards feeling “Damn Konfident” about the findings we obtain from verbal report methods (Martens, 1987).

Don’t Know

Damn Konfident

Acknowledgements US Office of Naval Research, Arlington, VA

The Florida State University Center for Research and Creativity

Judd Butler, Stavroula Perdikogiannis, Jason Ramratten, Mike Tuffiash, Ning Zhuang

Pat Thomas Law Enforcement Academy, Quincy, FL