Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies
at UMUCIrmak Renda-Tanali, DSc.
Collegiate Associate ProfessorDirector, Homeland Security & Emergency Management
Graduate School of Management & TechnologyUniversity of Maryland University College
Fourteenth Annual Emergency Management Higher Education Conference Federal Emergency Management Agency, U.S. Department of Homeland Security
Thursday June 9, 2011
Outline• Background• Research Objectives• Methodology
• Research Design• Tools
• Results• Survey Results• Actual Performance Results
• Conclusions• Recommendations
2Irmak Renda-Tanali
Background• Why such study?
• Emergency Management and related fields are best learned through scenario studies
• Textbooks and lecture-based learning do not effectively teach:
• Consensus gathering• Collective decision making• Decision making under uncertainty • Testing and validating assumptions • How to manage major crises and emergencies
• Textbooks and lectures do not capture the dynamic, time-sensitive, context-dependent, multi-disciplinary nature of the emergency/crisis scenarios. 3Irmak Renda-Tanali
Background• Why such study (cont’d)?
• Course evaluations and teaching experience show:
• Students want dynamic, hands-on exercises that simulate real world environments, real world thinking.
• Textbook reading and conference participations need further stimulating tools to enhance student understanding of topics.
• Students want to be an active part of decision making process as crisis events unfold.
• They want to learn what are the good AND bad decisions and why, by actively seeing consequences of their decisions in reaction to events that unfold.
• Case studies looking backward are not so interesting as the events and consequences are already known.
• Students learn better by hands-on practice than just by reading.
4Irmak Renda-Tanali
Background• Examples of current practice:
• Multi-media technology • Video, audio, web pages, web 2.0 tools: blogs,
wikis, social networking sites, podcasts, vodcasts etc.
• Play2Train, a virtual world platform, uses Second Life software with avatars (intelligent agents) mimicking human behavior.
• Specific examples related to security studies:
• A web-based scenario simulation tool used for public health undergraduate students (Spinello & Fishbach, 2004)
• San Luis Rey town scenario simulation exercise for HS students at UMUC graduate studies (Boubsil & Gayol, 2006)
• Play2Train, used to replicate tabletop exercises, e.g. Centers for Disease Control (CDC), Emory University, University of Illinois at Chicago’s Center for the Advancement of Distance Education (CADE), Seton Hall University, and Idaho State University (Hewitt et al, 2009)
5Irmak Renda-Tanali
Background• Examples (cont’d)
• Specific examples related to security studies (cont’d)
• Second Life and Play2Train in Master of Healthcare Administration online programs, used to enhance student discovery, critical thinking, and analytical skills.
6Irmak Renda-Tanali
Research Objectives• To capture the effectiveness of online
scenario simulation tools used at UMUC, on student learning;
• To identify the kinds of strengths and weaknesses of those tools;
• To compare those tools to traditional methods of teaching;
• Ultimately to enhance student learning in Emergency Management and related fields of study through the findings of the study.
7Irmak Renda-Tanali
Study Tools• Subjects:
• 5 online sections of CCJS 320 – 145 students in total
• 3 online sections of CCJS 420 – 78 students• 2 online sections of CCJS 425 – 45 students • 2 online sections of BSBD 641 – 51 students
total 319 students• Instructors• Developer-instructorsTaught in Fall 2010 semester
• Data Sources• Student survey for all sections of the above
courses• Phone calls to faculty developed and/or use
simulation• Course statistics measuring student
participation and interaction• Gradebook• The generic end-of-semester evaluations
8Irmak Renda-Tanali
Areas Surveyed• Learning:
• Understanding the material• Processing, testing, validating
assumptions• Decision making skills• Getting close to real-life experience• Connecting events and objects (or
actions)• Visualizing, conceptualizing,
comprehending• Consensus gathering• Effective crisis communication• Effective use of resources• Effective emergency response to large
scale crises• Student-to-student interaction• Student-to-faculty interaction
9Irmak Renda-Tanali
Assumptions• Learning:
• Online scenario simulation tools help understand the material better.
• They help better process, test, validate assumptions
• Improve decision making skills.• Help get close to real-life experience.• Help connect events and objects (or
actions).• Help with visualizing, conceptualizing,
and comprehending.• Increased student-to-student
interaction contributes to learning (Community of Inquiry Theory by Garrison et. al, 2000)
• Increased student-to-faculty interaction contributes to learning (CoI).
• Simulation tool increased interaction learning
11Irmak Renda-Tanali
Assumptions• Technology use:
• If there are no problems with the technology, student satisfaction is more better learning
• Workload:• No assumptions about the time spent
versus contribution to learning
• Student Satisfaction:• Students who responded favorably
performed favorably – or did they?
• Demographics:• No assumptions – a correlation study
may uncover unknown facts
12Irmak Renda-Tanali
Setting CCJS 320– Project Name: “Crime Scene Simulation”– Actions: Students virtually move around a
crime scene, examine items, process evidence, place them in an evidence box, interview the witness, and record details in a journal. After all the evidence is collected, students enter into a virtual Crime Lab and request the appropriate tests for each evidence item such as medico-legal autopsy, fingerprint analysis, forensic biology, toxicology, firearm examination, etc.
– Deliverable: After the tests are performed a detailed Crime Scene Report is generated. Students are graded based on the correctness of the evidence collected and the tests generated. The crime lab yields no results if incorrect tests are requested.
13Irmak Renda-Tanali
Results: Crime Scene Simulation• Response Rates (CCJS 320):
18Irmak Renda-Tanali
CCJS 320 No. of Surveys sent
No. of CompletedSurveys
Response rate
6380 32 12 37.5%
6381 30 9 30.0%
6980 29 14 48.3%
6981 25 11 44.0%
7980 29 5 17.2%
Total 145 51 35.2%
• Understanding the material
19Irmak Renda-Tanali
Strongly Disagree Disagree Agree Strongly Agree
0%
16%
43%41%
This simulation exercise helped me understand the material better. Question Q. No. Q2understanding material Q2 1.000hands-on experience Q3 0.788mirroring real-life exper. Q4 0.683improve decision making Q5 0.747assumption testing Q6 0.821making connections Q7 0.838visualizing, conceptualizing Q8 0.746engaging w/other students Q9 0.364engaging w/faculty Q10 0.424approaching faculty Q11 0.247technology (no bugs) Q12 0.444time spent Q13 -0.026no. of classes taken so far Q19 -0.271age Q22 -0.165
Large correlation(Pearson Coefficient)
Results: Crime Scene Simulation
Correlation to itself
Large (strong) correlation
Medium correlation
Low correlation
20Irmak Renda-Tanali
Pearson’s Correlation Coefficient
Pearson's correlation coefficient between two variables is defined as the Covariance of the two variables divided by the product of their standard deviations.Simple to do using Excel spreadsheet with the embedded function.
Correlation Negative Positive
None −0.09 to 0.0 0.0 to 0.09
Small −0.3 to −0.1 0.1 to 0.3
Medium −0.5 to −0.3 0.3 to 0.5
Large −1.0 to −0.5 0.5 to 1.0
Results Summary: Crime Scene Simulation
Learning AGREE CORRELATION
This simulation exercise helped me understand the material better. 84% LARGE
It gave me a hands-on opportunity to understand the material. 84% LARGE
It deepened my understanding of the subject as it mirrored real life situations. 84% LARGE
It helped to improve my decision making skills. 82% LARGE
It improved my skills in processing, testing, and validating assumptions.
80% LARGE
It improved my skills about making connections between events and objects.
80% LARGE
It helped me visualize, conceptualize, and comprehend situations related to crime scene investigation.
88% LARGE
21
Results Summary: Crime Scene Simulation
22
Interaction AGREE CORRELATION
The simulation helped me engage with other students.
37%MEDIUM (decision making,
understanding, technology)
LOW on all others
It improved my interaction with the instructor.
76% MEDIUM
It gave me ideas for questions about the subject matter to approach the instructor. 78% LOW - MEDIUM
Results Summary: Crime Scene Simulation
23
Other/DemographicsAGREE CORRELATION
Technology worked 74% MEDIUM
Time spent Diverse (2 to 8 hours) MEDIUM (approaching faculty), low
otherwise
# of classes taken 61% more than 4 LOW NEGATIVE on all, no correlation with
interaction
Age49% between 20-30 years,
27% between 30-40 yrs
MEDIUM (NEGATIVE: mirroring real-life;
improve decision making; assumption testing;
making connections); LOW (NEGATIVE: understanding, hands-on exp., visualizing);
LOW (POSITIVE: engaging w/students &
faculty, no of classes taken)
Gender 69% female N/A
Results Summary: Crime Scene Simulation
24
Standard Course Evaluation Results (related questions) (Mean out of 5-point Likert Scale)
Technology (such as slide shows, multi-media, streaming audio-video) was effectively used in this course 4.35
This course helped me develop or improve my computer skills.3.85
The lab activities contributed to my learning.4.15
I was able to use technology effectively on my own.4.36
This course enhanced student-to-student interaction.4.17
This course enhanced faculty-student interaction.4.06
Setting CCJS 420• CCJS 420 Medical and Legal
Investigations of DeathCold Case Simulation
• Actions: Students begin by examining the reports and evidence from the original investigation. Once they are familiar with the homicide case and the people involved, they proceed to interview persons of interest. As the simulation progresses and student encounter new evidence, they will be able to submit that evidence for various types of forensic testing.
• As students proceed with their investigation, they gather evidence, test results, reports, and witness statements. These items are placed into their binder as they are created or discovered.
Deliverable: Arrest Warrant ReportObjectives: Gather enough evidence, secure
an indictment from the district attorney, and solve the case.
25Irmak Renda-Tanali
• Response Rates (CCJS 420):
27Irmak Renda-Tanali
CCJS 420No. of Surveys sent
No. of Responses
Response rate section
6980 27 12 44.4%
7380 19 6 31.6%
7980 30 8 26.7%
Total 76 26 34.2%
Results: Cold Case File Simulation
Results Summary: Cold Case File Simulation
Learning AGREE CORRELATION
This simulation exercise helped me understand the material better. 84% LARGE
It gave me a hands-on opportunity to understand the material. 88% LARGE
It deepened my understanding of the subject as it mirrored real life situations. 68% LARGE
It helped to improve my decision making skills. 75% LARGE
It improved my skills in processing, testing, and validating assumptions.
75% LARGE
It improved my skills about making connections between events and objects.
84% LARGE
It helped me visualize, conceptualize, and comprehend situations related to crime scene investigation.
92% LARGE
28
Results Summary: Cold Case File Simulation
29
Interaction AGREE CORRELATION
The simulation helped me engage with other students.
40%
LARGE (making connections; visualizing and
conceptualizing; engaging & approaching faculty);
MEDIUM (hands-on, mirroring, decision making,
assumption testing);
LOW (understanding, technology); LOW (negative: time
spent and age)
It improved my interaction with the instructor. 44%
LARGE (learning)
MEDIUM (technology, negative: age);
LOW (negative: time spent)
It gave me ideas for questions about the subject matter to approach the instructor. 71%
LARGE (hands-on, real-life, making connections,
visualizing, engaging),
MEDIUM (understanding, decision, assumption testing,
technology), MEDIUM (negative: age), LOW (negative: time spent)
Results Summary: Cold Case File Simulation
30
Other/DemographicsAGREE CORRELATION
Technology worked 56%
LARGE (hands-on, real-life), MEDIUM
(understanding, decision-making, assumption testing, making connections, engaging/
approaching faculty), MEDIUM (NEGATIVE: age); LOW (visualizing,
engaging w/students, no. of classes)
Time spent Average
NEGATIVELARGE (NEGATIVE: visualizing, age),
MEDIUM (NEGATIVE: hands-on, real-
life, assumption testing, making connection),
LOW (NEGATIVE: understanding,
interaction)
# of classes taken92% more than 4 LOW
Age44% between 20-30 years,
44% between 30-40 yrs
LARGE (time spent), MEDIUM
(NEGATIVE: understanding, assumption
testing, visualizing, interaction), LOW
(NEGATIVE: hands-on, decision-making,
making connections, engaging w/other students, no. of classes taken)
Gender88% female N/A
Results Summary: Cold Case File Simulation
31
Standard Course Evaluation Results (related questions) (Mean out of 5-point Likert Scale)
Technology (such as slide shows, multi-media, streaming audio-video) was effectively used in this course 3.68
This course helped me develop or improve my computer skills.4.15
The lab activities contributed to my learning.4.04
I was able to use technology effectively on my own.4.39
This course enhanced student-to-student interaction.4.04
This course enhanced faculty-student interaction.3.50
Results: Biological Attack Simulation
• Response Rates (BSBD 641):
33Irmak Renda-Tanali
BSBD 641section
No. of Surveys sent
No. of Responses
Response rate
9040 26 9 34.6%
9041 25 9 36.0%
Total 51 18 35.3%
Results Summary: Biological Attack Simulation
Learning AGREE CORRELATION
This simulation exercise helped me understand the material better.
100% LARGE (hands-on, real life, decision making, eff. Crisis
comm.., approaching faculty and technology), MEDIUM (eff. Em response, resources) LOW (consensus)
It gave me a hands-on opportunity to understand the material.
100% LARGE (understanding, real-life, crisis comm..),
MEDIUM (decision, eff. EM response, consensus,
resources)
It deepened my understanding of the subject as it mirrored real life situations.
89% LARGE (understanding, hands-on, decision-making, eff.
Crisis comm.), MEDIUM (eff. EM response, consensus,
resources)
It helped to improve my decision making skills. 89% LARGE (understanding, real-life, consensus, eff. Crisis
comm.), MEDIUM (hands-on, eff. EM response,
resources)
It improved my knowledge of establishing an effective emergency response structure during bioterrorism attacks.
100% LARGE (hands-on, real life, decision making, eff. Crisis
comm.., approaching faculty and technology), MEDIUM (eff. Em response, resources) LOW (consensus)
It improved my consensus gathering skills. 100% LARGE (consensus, resources, eff. Crisis comm.),
MEDIUM (understanding, hands-on, real-life, decision
making)
It helped me understand what resources are needed during a public health bioterrorism emergency.
94% LARGE (decision making, eff. EM response, resources, eff.
Crisis comm.) MEDIUM (hands-on, real-life), LOW
(understanding)
It helped improve my understanding of effective crisis communication.
100% LARGE (eff. Em response, consensus, eff. Crisis comm..)
MEDIUM (understanding, hands-on, real-life, decision
making)
34
Results Summary: Cold Case File Simulation
35
Interaction AGREE CORRELATION
The simulation helped me engage with other students.
94%
LARGE (consensus, eff Crisis comm.) MEDIUM (understanding, hands-on, decision making, eff. EM
response, resources, interaction w/faculty), LOW (real-life)
LOW (NEGATIVE: time spent, no. of classes taken)
It improved my interaction with the instructor. 50%
LARGE (decision making, consensus gathering, eff. Crisis
comm., approaching faculty) MEDIUM (understanding,
hands-on, real-life, eff. EM response, resources, engaging
with students) LOW (age), LOW (negative: no. of classes)
It gave me ideas for questions about the subject matter to approach the instructor. 71%
LARGE (understanding, real-life, decision making,
consensus gathering, eff. Crisis comm.., engaging with
faculty, technology) MEDIUM (hands-on, eff. EM
response, resources, engaging with students), LOW
(NEGATIVE: time spent) LOW (no of classes)
Results Summary: Cold Case File Simulation
36
Other/DemographicsAGREE CORRELATION
Technology worked 89%
LARGE (understanding, hands-on, real-life,
decision making, effective crisis comm.., engaging w/ students, approaching faculty)
MEDIUM (consensus, engaging with faculty,
time spent) LOW (eff EM response, resources)
LOW (NEGATIVE: age)
Time spent 67 % More than 8 hoursNEGATIVE MEDIUM (real-life,
technology), LOW ON OTHERS
# of classes taken 89% more than 4 MEDIUM (hands-on) LOW on others
Age22% 20-30 yrs, 39% 30-40 yrs,
22% 41-50, 17% 50-60 yrs
MEDIUM (negative: understanding), LOW (NEGATIVE: real-life, time spent) LOW (decision making, eff crisis comm., engaging w/faculty, no of classes)
Gender 17% female N/A
Results Summary: Cold Case File Simulation
37
Standard Course Evaluation Results (related questions) (Mean out of 5-point Likert Scale)
Technology (such as slide shows, multi-media, streaming audio-video) was effectively used in this course 3.92
This course helped me develop or improve my computer skills.3.34
The lab activities contributed to my learning.4.22
I was able to use technology effectively on my own.4.21
This course enhanced student-to-student interaction.4.61
This course enhanced faculty-student interaction.3.71
Results Summary
Crime Scene Cold case BioattackAGREE CORRELATIO
NAGREE CORRELATION AGREE CORRELATION
LearningUnderstanding material
84% LARGE 84% LARGE 100% LARGE (hands-on, real life,
decision making, eff. Crisis comm.., approaching faculty and
technology), MEDIUM (eff. Em
response, resources) LOW (consensus)
Hands-on experience
84% LARGE 88% LARGE 100% LARGE (understanding, real-life,
crisis comm..), MEDIUM (decision, eff. EM response, consensus, resources)
Mirroring real-life experience
84% LARGE 68% LARGE 89% LARGE (understanding, hands-
on, decision-making, eff. Crisis
comm.), MEDIUM (eff. EM
response, consensus, resources)
Improving decision making skills
82% LARGE 75% LARGE 89% LARGE (understanding, real-life,
consensus, eff. Crisis comm.),
MEDIUM (hands-on, eff. EM
response, resources)
38
Results SummaryCrime Scene Cold case Bioattack
AGREE CORRELATION
AGREE CORRELATION AGREE CORRELATION
LearningValidating and testing assumptions
80% LARGE 75% LARGE N/A N/A
Making connections between events and objects
80% LARGE 84% LARGE N/A N/A
Helps visualizing, conceptualizing
88% LARGE 92% LARGE N/A N/A
Understanding effective emergency response
N/A N/A N/A N/A 100%
LARGE (consensus, resources,
eff. Crisis comm.), MEDIUM (understanding, hands-on, real-life, decision making)
Improve consensus gathering skills N/A N/A N/A N/A 94%
LARGE (decision making, eff.
EM response, resources, eff.
Crisis comm.) MEDIUM (hands-on, real-life), LOW
(understanding)
Learn effective use of resources in EM response
N/A N/A N/A N/A 100%
LARGE (eff. Em response,
consensus, eff. Crisis comm..)
MEDIUM (understanding,
hands-on, real-life, decision making)
Improve effective crisis communication skills
N/A N/A N/A N/A 94% LARGE
39
Results Summary
40
Crime Scene Cold case BioattackAGREE CORRELATIO
NAGREE CORRELATION AGREE CORRELATION
Interaction
Student to student engagement
37%
MEDIUM
(decision making, understanding,
technology) LOW on all others
40%
LARGE (making
connections; visualizing and conceptualizing;
engaging & approaching faculty);
MEDIUM (hands-
on, mirroring, decision making, assumption
testing); LOW (understanding,
technology); LOW
(negative: time spent and age)
94%
LARGE (consensus, eff Crisis
comm.) MEDIUM (understanding, hands-on, decision making, eff. EM response, resources, interaction
w/faculty), LOW (real-life)
LOW (NEGATIVE: time
spent, no. of classes taken)
Engaging with faculty
76%
MEDIUM
44%
LARGE (learning)
MEDIUM
(technology, negative:
age); LOW (negative:
time spent)50%
LARGE (decision making,
consensus gathering, eff. Crisis comm., approaching faculty)
MEDIUM (understanding,
hands-on, real-life, eff. EM response, resources, engaging
with students) LOW (age),
LOW (negative: no. of classes)
Approaching faculty
78%
LOW - MEDIUM
71%
LARGE (hands-on,
real-life, making connections,
visualizing, engaging),
MEDIUM
(understanding, decision, assumption testing, technology),
MEDIUM (negative:
age), LOW (negative:
time spent)
71%
LARGE (understanding, real-
life, decision making, consensus gathering, eff. Crisis comm.., engaging with faculty, technology)
MEDIUM (hands-on, eff. EM
response, resources, engaging
with students), LOW
(NEGATIVE: time spent)
LOW (no of classes)
Results Summary
41
Crime Scene Cold case BioattackAGREE CORRELATIO
NAGREE CORRELATION AGREE CORRELATION
Other issuesTechnology worked
74%
MEDIUM
56%
LARGE (hands-on,
real-life), MEDIUM
(understanding, decision-making,
assumption testing, making connections,
engaging/ approaching
faculty), MEDIUM (NEGATIVE: age);
LOW (visualizing,
engaging w/students, no. of classes)
89%
LARGE (understanding, hands-
on, real-life, decision making, effective crisis comm.., engaging w/ students, approaching faculty)
MEDIUM (consensus,
engaging with faculty, time spent)
LOW (eff EM response,
resources) LOW
(NEGATIVE: age)
Time spent
Uniformly distributed
MEDIUM (approaching faculty), low otherwise
Average
NEGATIVELARGE
(NEGATIVE: visualizing, age),
MEDIUM
(NEGATIVE: hands-
on, real-life, assumption testing, making
connection), LOW
(NEGATIVE: understanding,
interaction)
67 % More than 8 hours
NEGATIVE MEDIUM (real-life, technology), LOW
ON OTHERS
# of classes taken 61% more
than 4
LOW NEGATIVE on
all, no correlation with interaction
92% more than 4
LOW
89% more than 4
MEDIUM (hands-on) LOW on others
Results Summary
42
Crime Scene Cold case BioattackAGREE CORRELATIO
NAGREE CORRELATION AGREE CORRELATION
DemographicsAge 49%
between 20-30
years, 27% between 30-40 yrs
MEDIUM (NEGATIVE: mirroring real-life; improve decision
making; assumption
testing; making connections);
LOW
(NEGATIVE: understanding, hands-on exp.,
visualizing);
LOW
(POSITIVE: engaging
w/students & faculty, no of
classes taken)
44% between 20-30 years, 44% between
30-40 yrs
LARGE (time spent),
MEDIUM
(NEGATIVE: understanding,
assumption testing, visualizing, interaction),
LOW (NEGATIVE: hands-on, decision-
making, making connections, engaging w/other students, no. of
classes taken)
22% 20-30 yrs, 39% 30-40 yrs, 22% 41-50,
17% 50-60 yrs
MEDIUM (negative:
understanding), LOW (NEGATIVE: real-life, time
spent) LOW (decision making,
eff crisis comm., engaging w/faculty, no of classes)
Gender 69% female 88% female 17% female
Conclusions
43
• Online scenario simulations add value to understanding.
• They help learning through discovery.• Provide near real-life, hands-on
experience.• Help visualize, conceptualize and make
connections between events and tools/objects.
• There is a correlation between student satisfaction and grades. Further studies with control group settings can uncover this or more facts.
• May or may not increase interaction among students. Further studies should look into this.
Conclusions
44
• There is evidence of increased student-to-faculty interaction.
• Technology glitches diminish value and satisfaction. (e.g. dead ends, loops)
• Information glitches diminish value and satisfaction.
• Age plays an important role in designing the content for the student audience.
• A negative correlation between time spent and learning the material was uncovered. Was the time spent on technical problems? Further studies should look into optimal use of time. Too much time spent may bring diminishing returns.
Conclusions
45
• The more sophisticated (game like) features make the tool more satisfactory.
• Students would like a sense of achievement at the end (e.g. solving piece of a puzzle, putting pieces together)
• Caution: Make them game-like but emphasize pedagogy.
• If designed as a group work with role playing ability may increase communication skills, help build core competencies in crisis management such as consensus gathering, effective resource allocation, understanding response structures in large scale crises and disasters.
Conclusions
46
• Recommend a move from low budget productions to sophisticated online simulation tools if budget permits.
• Expanded studies may show a link between the dollar amount spent for development and the student success outcome (Benefit-Cost Analysis).
• The tools should allow room for changing/ alternating between multiple scenarios to prevent boredom among professors and student sharing of results.
• Need buy-in from instructors. Engage them in designing the tools.
Conclusions
47
• There are successful applications with proven results about other similar simulations employed by other higher ed instutions. UMUC can emulate on that.
• The new generation as “Digital natives” learn better with game-like simulations.