Development of a Situational Judgment Test for Teamwork in

MedicineThesis Defense for Tommy May

Central Michigan University

Department of Psychology

Introduction Teamwork is vital in the medical community, a report by the

Institute of Medicine finds that the majority of accidents in medical facilities are due to a break down in teamwork (Kohn, Corrigan, & Donaldson, 1999)

Currently, teamwork is assessed using behavioral checklist (which is costly) and self report (which is not very accurate in determining abilities) (DeNisi & Shaw, 1977).

Situational Judgment Tests offer a more cost effective way to assess an individuals ability to apply knowledge of a subject (Christian, Edward, and Bradley, 2010).

The current study addresses this need for a more effective measure of teamwork through the development of a Situational Judgment Test.

Teamwork dimensions were based off of the research used to develop and implement the TeamSTEPPS training program. The teamwork dimensions are:

Communication – The sharing of task-important information with the team, including information about patient condition and medical history, actions one should be taking, has taken, or are currently taking.

Team Structure – Knowing the responsibilities of yourself and other team members and ensuring that your responsibilities are performed according to standard operation procedure.

Teamwork Dimensions

Mutual Support - Making team members aware of their mistakes/performance, helping others learn from mistakes, or appropriate reactions to errors in treatment. Helping other team members perform their physical tasks; offering social/emotional support to upset or frustrated team members.

Leadership - Providing clear direction, initiating structure, and/or taking command of a chaotic situation. It also includes appropriately handling objections or suggestions from subordinates.

Environmental Support - How cognizant team members are of the medical conditions of the patient, the supplies, resources, and current demand in the emergency department, and what actions other team members have taken.

Teamwork Dimensions, cont.

Assessment of Teamwork Assessment of Content

Skill-Based: motor or technical skills

Behavior Observation/Checklist

Affective-Based: change in attitude or motivation

Self-Report

Cognitive-Based: knowing how to acquire, apply, and organize knowledge of content

Not readily assessed in Teamwork among Medicine

Who is to be Assessed

The team as a whole

Individuals within the team

Emergency room teams frequently change

Competencies of interest are defined and frequently trained at the individual-level

(Kraiger et al., 1993)

Situational Judgment Test (SJT)

Describe a situation in which respondents must select the appropriate action to take Test knowledge of construct

Test application of knowledge

Carry greater degree of job relevance Situations are developed using real examples collected from

experienced professionals

SJT in Teamwork and Medicine SJT’s to assess Teamwork

Predict team member role performance above and beyond general mental ability (Mumford et al., 2008)

SJT’s measuring teamwork (r=.38) and leadership (r=.28) have demonstrated modest relationships with job performance (Christian et al., 2010)

SJT’s in the Medical Field

SJT’s in selection for medical school candidates predicted above and beyond general mental ability (Lievens et al., 2005 & Patel et al., 2012)

SJT’s have been found to be effective at assessing interpersonal skills (Lievens, 2013’ Lievens & Sackett, 2012; & Patterson et al., 2012)

Method Collect examples of effective and ineffective teamwork

Using an online survey, experts in emergency medicine gave examples of effective and ineffective teamwork. Describing why they felt the example was or was not effective.

Developed items based off of examples The examples provided by professionals were used to

develop situational judgment items. Items were written with the intent to fit in to one of the 5

teamwork dimensions Sort items in to teamwork dimensions

Items were sorted into the five dimensions by a 4 professionals Remove items with low agreement

Items that had low agreement were removed to ensure that each item was measuring the intended dimension

Rater Agreement

Table 1. Inter-rater Agreement

Raters Kappa1 and 2 11 and 3 0.881 and 4 0.52 and 3 0.882 and 4 0.53 and 4 0.5Average 0.71

Results indicate that rater agreement was respectable with the average Kappa at .71

Table 2. Rater AgreementItem Number Item Dimension Agreement

1 Environmental Awareness 50 %2 Environmental Awareness 100%3 Communication 100%4 Communication 100%5 Team Structure 75%6 Team Structure 75%7 Mutual Support 100%8 Team Structure 100%9 Leadership 75%10 Environmental Awareness 100%11 Leadership 100%12 Communication 100%13 Environmental Awareness 50%14 Leadership 75%15 Leadership 100%16 Communication 100%17 Team Structure 75%18 Mutual Support 100%19 Mutual Support 75%20 Mutual Support 100%

Note. Percentage is computed from 4 raters.

Development of Alternate SJT Forms

Item isomorphism was used following Lievens and Sackett (2007)

radicals are features that determine the items difficulty

incidentals are only surface level characteristics that do not impact difficulty

In item isomorphism, the radicals are the domain of the stem, the critical incident, and the context of the incident. The incidentals are the language and grammar found in the stem or response options.

Hypothesis

Hypothesis 1A: Medical students’ scores on the T-TAQ will not significantly differ from medical professionals’ scores on the T-TAQ.

Hypothesis 1B: Medical students’ scores on the developed SJT will significantly differ from medical professionals’ scores on the developed SJT. Such that medical professionals will outperform the medical students.

Hypothesis 2: The developed SJT will have weak to modest correlations with the T-TAQ.

Hypothesis 3: Form A of the developed SJT will have a strong and positive correlation with Form B.

Procedures Medical Students recruited through CMED

Given $7 Amazon gift card for their help

Medical Professionals recruited through personal contacts and SEMPA

Given $20 Amazon gift card for their help

Survey order

Form A or B

T-TAQ

Form B or A

Few demographic items

Participants 75 participants volunteered to complete survey

17 participants removed

Could not compare forms A and B

Could not compare SJT and T-TAQ scores

Could not identify level of medical experience

58 complete surveys

Sample was 51.7% male and 81% Caucasian

35 individuals were medical professionals

23 were medical students

Developing Test Key Participants rank ordered each response option from 1-least effective

to 5-most effective

Mean ranking of the professionals response options were used as the correct ranking

Would bring into account more opinions of professionals than using the mode

Deviations score was computed from professionals’ mean ranking of response options

K – R = D the absolute value of D was used as the deviation score

K = the mean ranking of professionals

R = respondents ranking

The greater an individuals deviation score the poorer they performed on the measure

0 5 10 15 20 25 30 350

10

20

30

40

50

60

70

Professional's SJT Score

Years Experience in Emergency Department

SJT

Devi

ati

on S

core

Test ReliabilityTable 3. Correlations Between Test Forms by Item.

Item Construct Correlations with Form B Item

1 Environmental Awareness 1 .682 Environmental Awareness 2 .623 Communication 1 .524 Communication 2 .695 Team Structure 1 .626 Team Structure 2 .427 Mutual Support 1 .528 Team Structure 3 .549 Leadership 1 .6210 Environmental Awareness 3 .4411 Leadership 2 .4812 Communication 3 .5913 Environmental Awareness 4 .6514 Leadership 3 .5415 Communication 4 .6216 Leadership 4 .3117 Team Structure 4 .4618 Mutual Support 2 .6019 Mutual Support 3 .6320 Mutual Support 4 .61

Note. Items in bold were later removed

Table 5. Correlations Among SJT Forms Total Scores.  1 2 3 41. Form A All Items        2. Form A Items removed

.99      

3. Form B All items .87 .87    4. Form B Items removed

.87 .88 .98  

The high correlation between forms (r =.88) and the great correlations between items suggest that the measure is reliable, supporting hypothesis 3.

Comparing the SJT and T-TAQTable 6. Correlations Between T-TAQ and SJT by Constructs and Total Scores.  1 2 3 4 5 6 7 8 9 10 11 12

1. Team Structure TAQ                        

2. Leadership TAQ  .71                      

3. Situational Monitory TAQ

.76 .72                    

4. Mutual Support TAQ .43 .48 .31                  

5. Communication TAQ .67 .57 .54 .49                

6. Team Structure SJT -.05 -.15 -.14 -.27 -.17              

7. Leadership SJT .00 -.10 -.03 -.14 -.04 .51            

8. Environmental Awareness SJT

-.12 -.26 -.19 -.41 -.28 .63 .39          

9. Mutual Support SJT -.19 -.20 -.15 -.33 -.36 .46 .40 .50        

10. Communication SJT -.11 -.08 -.18 -.31 -.13 .48 .39 .44 .35      

11. TAQ total .88 .87 .84 .69 .82 -.19 -.11 -.31 -.32 -.21    

12. SJT total  -.12 -.22 -.19 -.39 -.26 .84 .69 .82 .84 .71 -.30  Results support hypothesis 2

Performance on the SJT and T-TAQ

Table 7. Means and Standard Deviation for T-TAQ Scores Between Professionals and Students.  Medical

StudentsMedical

Professionals 

Mean SD Mean SD Effect SizeTeam Structure TAQ 4.38 .33 4.26 .67 -.21Leadership TAQ 4.45 .38 4.56 .59 .21Situational Monitory TAQ

4.22 .53 4.41 .61 .33

Mutual Support TAQ 4.22 .51 4.35 .54 .25Communication TAQ 4.10 .51 4.16 .59 .11T-TAQ Total 21.24 1.59 21.72 2.67 .21

To test hypothesis 1A, a one-tailed t-test demonstrated that the total score on the T-TAQ between medical students (M= 21.24 SD=1.59) and medical professionals (M=21.71 SD=2.67) was not significant, t(50)= -.74, p= .233. Results support hypothesis 1A.

Table 8. Means and Standard Deviation for SJT Scores Between Professionals and Students.

Item Medical Students Medical Professionals Effect Size 95% Confidence Interval

Mean SD Mean SD      

1 5.44 1.65 4.56 1.29 -.61 -1.15 -.072 3.55 2.20 2.94 1.73 -.32 -.85 .213 2.96 1.91 1.65 1.07 -.9 -1.45 -.355 4.81 2.18 3.54 1.66 -.68 -1.22 -.136 3.21 1.89 2.35 1.09 -.59 -1.13 -.057 3.77 1.00 3.47 1.41 -.28 -.77 .298 5.64 1.95 4.15 1.10 -1.00 -1.55 -.449 3.85 2.16 3.07 1.62 -.42 -.95 .1110 3.95 2.68 2.62 1.21 -.69 -1.23 -.1511 3.75 1.64 3.23 1.04 -.4 -.93 .1313 4.61 2.22 2.73 1.24 -1.11 -1.67 -.5514 2.93 1.54 1.94 1.00 -.8 -1.34 -.2515 4.22 1.82 3.41 1.36 -.52 -1.05 .0116 3.48 2.69 1.90 1.07 -1.12 -1.68 -.5517 3.93 1.71 2.81 1.34 -.74 -1.29 -.2119 4.33 1.92 2.83 0.90 -1.08 -1.64 -.5220 2.91 1.57 2.45 1.17 -.34 -.87 .19Total 67.34 17.36 49.65 4.88 -1.54 -2.13 -.94

Table 9. Means and Standard Deviations for Total and Average Scores for the SJT and T-TAQ

  Medical Students Medical Professionals

Means SD Means SD

SJT Total 67.34 17.36 49.65 4.88

SJT Average 3.96 1.02 2.92 .29

T-TAQ Total 21.24 1.59 21.72 .32

T-TAQ Average 4.25 2.67 4.34 .53

A one-tailed t-test demonstrates that medical students had a significantly higher deviation score (M=67.34 SD= 17.36) on the SJT than medical professionals (M=49.65 SD=4.88), t(56)=5.72, p< .01, supporting hypothesis 1B.

SJT Measure0

10

20

30

40

50

60

70

80

SJT Performance

Students Professionals

Limitations

Low sample size, leaves the study vulnerable to error and extraneous variables

Representativeness of the sample

All medical students were selected from one university

Would other medical professionals (i.e. nurses, anesthesiologist, etc.) respond in the same way?

A more thorough validation

Using teamwork training

Concurrent validity study

Predicting success in teamwork training courses

Discussion

The T-TAQ is best used for assessing team members attitudes towards teamwork when team members have little reason to distort their

responses.

SJT assess teamwork knowledge, students and professionals differ with regards to medical knowledge and teamwork knowledge.

Further validation of the SJT measure is needed, ideally testing a group of medical students before and after teamwork training would

be a great start.

With the measure further validated it could be used in assessing training needs in medical facilities, in employee hiring for medical professionals, and even selecting medical students into a program.

Questions?

Thank you

A big thank you to the many professionals and students who dedicated their time and energy to helping complete this project.

Specifically:

Dr. Matthew Prewett, Committee Chair & Advisor

Dr. Neil Christiansen and Dr. Steven Vance, Committee Members

Spencer Thronock, CMED student

Most importantly

Chelsea (my wife), Zach, & Harvey (my boys), for motivation