Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies at UMUC Irmak Renda-Tanali, DSc. Collegiate Associate Professor

Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies

at UMUCIrmak Renda-Tanali, DSc.

Collegiate Associate ProfessorDirector, Homeland Security & Emergency Management

Graduate School of Management & TechnologyUniversity of Maryland University College

Fourteenth Annual Emergency Management Higher Education Conference Federal Emergency Management Agency, U.S. Department of Homeland Security

Thursday June 9, 2011

Outline• Background• Research Objectives• Methodology

• Research Design• Tools

• Results• Survey Results• Actual Performance Results

• Conclusions• Recommendations

2Irmak Renda-Tanali

Background• Why such study?

• Emergency Management and related fields are best learned through scenario studies

• Textbooks and lecture-based learning do not effectively teach:

• Consensus gathering• Collective decision making• Decision making under uncertainty • Testing and validating assumptions • How to manage major crises and emergencies

• Textbooks and lectures do not capture the dynamic, time-sensitive, context-dependent, multi-disciplinary nature of the emergency/crisis scenarios. 3Irmak Renda-Tanali

Background• Why such study (cont’d)?

• Course evaluations and teaching experience show:

• Students want dynamic, hands-on exercises that simulate real world environments, real world thinking.

• Textbook reading and conference participations need further stimulating tools to enhance student understanding of topics.

• Students want to be an active part of decision making process as crisis events unfold.

• They want to learn what are the good AND bad decisions and why, by actively seeing consequences of their decisions in reaction to events that unfold.

• Case studies looking backward are not so interesting as the events and consequences are already known.

• Students learn better by hands-on practice than just by reading.

4Irmak Renda-Tanali

Background• Examples of current practice:

• Multi-media technology • Video, audio, web pages, web 2.0 tools: blogs,

wikis, social networking sites, podcasts, vodcasts etc.

• Play2Train, a virtual world platform, uses Second Life software with avatars (intelligent agents) mimicking human behavior.

• Specific examples related to security studies:

• A web-based scenario simulation tool used for public health undergraduate students (Spinello & Fishbach, 2004)

• San Luis Rey town scenario simulation exercise for HS students at UMUC graduate studies (Boubsil & Gayol, 2006)

• Play2Train, used to replicate tabletop exercises, e.g. Centers for Disease Control (CDC), Emory University, University of Illinois at Chicago’s Center for the Advancement of Distance Education (CADE), Seton Hall University, and Idaho State University (Hewitt et al, 2009)

5Irmak Renda-Tanali

Background• Examples (cont’d)

• Specific examples related to security studies (cont’d)

• Second Life and Play2Train in Master of Healthcare Administration online programs, used to enhance student discovery, critical thinking, and analytical skills.

6Irmak Renda-Tanali

Research Objectives• To capture the effectiveness of online

scenario simulation tools used at UMUC, on student learning;

• To identify the kinds of strengths and weaknesses of those tools;

• To compare those tools to traditional methods of teaching;

• Ultimately to enhance student learning in Emergency Management and related fields of study through the findings of the study.

7Irmak Renda-Tanali

Study Tools• Subjects:

• 5 online sections of CCJS 320 – 145 students in total

• 3 online sections of CCJS 420 – 78 students• 2 online sections of CCJS 425 – 45 students • 2 online sections of BSBD 641 – 51 students

total 319 students• Instructors• Developer-instructorsTaught in Fall 2010 semester

• Data Sources• Student survey for all sections of the above

courses• Phone calls to faculty developed and/or use

simulation• Course statistics measuring student

participation and interaction• Gradebook• The generic end-of-semester evaluations

8Irmak Renda-Tanali

Areas Surveyed• Learning:

• Understanding the material• Processing, testing, validating

assumptions• Decision making skills• Getting close to real-life experience• Connecting events and objects (or

actions)• Visualizing, conceptualizing,

comprehending• Consensus gathering• Effective crisis communication• Effective use of resources• Effective emergency response to large

scale crises• Student-to-student interaction• Student-to-faculty interaction

9Irmak Renda-Tanali

Areas Surveyed• Technology use• Workload• Student Satisfaction• Demographics

10Irmak Renda-Tanali

Assumptions• Learning:

• Online scenario simulation tools help understand the material better.

• They help better process, test, validate assumptions

• Improve decision making skills.• Help get close to real-life experience.• Help connect events and objects (or

actions).• Help with visualizing, conceptualizing,

and comprehending.• Increased student-to-student

interaction contributes to learning (Community of Inquiry Theory by Garrison et. al, 2000)

• Increased student-to-faculty interaction contributes to learning (CoI).

• Simulation tool increased interaction learning


Assumptions• Technology use:

• If there are no problems with the technology, student satisfaction is more better learning

• Workload:• No assumptions about the time spent

versus contribution to learning

• Student Satisfaction:• Students who responded favorably

performed favorably – or did they?

• Demographics:• No assumptions – a correlation study

may uncover unknown facts


Setting CCJS 320– Project Name: “Crime Scene Simulation”– Actions: Students virtually move around a

crime scene, examine items, process evidence, place them in an evidence box, interview the witness, and record details in a journal. After all the evidence is collected, students enter into a virtual Crime Lab and request the appropriate tests for each evidence item such as medico-legal autopsy, fingerprint analysis, forensic biology, toxicology, firearm examination, etc.

– Deliverable: After the tests are performed a detailed Crime Scene Report is generated. Students are graded based on the correctness of the evidence collected and the tests generated. The crime lab yields no results if incorrect tests are requested.


Setting CCJS 320 (cont’d)








Results: Crime Scene Simulation• Response Rates (CCJS 320):


CCJS 320 No. of Surveys sent

No. of CompletedSurveys

Response rate

6380 32 12 37.5%

6381 30 9 30.0%

6980 29 14 48.3%

6981 25 11 44.0%

7980 29 5 17.2%

Total 145 51 35.2%

• Understanding the material


Strongly Disagree Disagree Agree Strongly Agree

0%

16%

43%41%

This simulation exercise helped me understand the material better. Question Q. No. Q2understanding material Q2 1.000hands-on experience Q3 0.788mirroring real-life exper. Q4 0.683improve decision making Q5 0.747assumption testing Q6 0.821making connections Q7 0.838visualizing, conceptualizing Q8 0.746engaging w/other students Q9 0.364engaging w/faculty Q10 0.424approaching faculty Q11 0.247technology (no bugs) Q12 0.444time spent Q13 -0.026no. of classes taken so far Q19 -0.271age Q22 -0.165

Large correlation(Pearson Coefficient)

Results: Crime Scene Simulation

Correlation to itself

Large (strong) correlation

Medium correlation

Low correlation


Pearson’s Correlation Coefficient

Pearson's correlation coefficient between two variables is defined as the Covariance of the two variables divided by the product of their standard deviations.Simple to do using Excel spreadsheet with the embedded function.

Correlation Negative Positive

None −0.09 to 0.0 0.0 to 0.09

Small −0.3 to −0.1 0.1 to 0.3

Medium −0.5 to −0.3 0.3 to 0.5

Large −1.0 to −0.5 0.5 to 1.0

Results Summary: Crime Scene Simulation

Learning AGREE CORRELATION

This simulation exercise helped me understand the material better. 84% LARGE

It gave me a hands-on opportunity to understand the material. 84% LARGE

It deepened my understanding of the subject as it mirrored real life situations. 84% LARGE

It helped to improve my decision making skills. 82% LARGE

It improved my skills in processing, testing, and validating assumptions.

80% LARGE

It improved my skills about making connections between events and objects.

80% LARGE

It helped me visualize, conceptualize, and comprehend situations related to crime scene investigation.

88% LARGE

21


22

Interaction AGREE CORRELATION

The simulation helped me engage with other students.

37%MEDIUM (decision making,

understanding, technology)

LOW on all others

It improved my interaction with the instructor.

76% MEDIUM

It gave me ideas for questions about the subject matter to approach the instructor. 78% LOW - MEDIUM


23

Other/DemographicsAGREE CORRELATION

Technology worked 74% MEDIUM

Time spent Diverse (2 to 8 hours) MEDIUM (approaching faculty), low

otherwise

# of classes taken 61% more than 4 LOW NEGATIVE on all, no correlation with

interaction

Age49% between 20-30 years,

27% between 30-40 yrs

MEDIUM (NEGATIVE: mirroring real-life;

improve decision making; assumption testing;

making connections); LOW (NEGATIVE: understanding, hands-on exp., visualizing);

LOW (POSITIVE: engaging w/students &

faculty, no of classes taken)

Gender 69% female N/A


24

Standard Course Evaluation Results (related questions) (Mean out of 5-point Likert Scale)

Technology (such as slide shows, multi-media, streaming audio-video) was effectively used in this course 4.35

This course helped me develop or improve my computer skills.3.85

The lab activities contributed to my learning.4.15

I was able to use technology effectively on my own.4.36

This course enhanced student-to-student interaction.4.17

This course enhanced faculty-student interaction.4.06

Setting CCJS 420• CCJS 420 Medical and Legal

Investigations of DeathCold Case Simulation

• Actions: Students begin by examining the reports and evidence from the original investigation. Once they are familiar with the homicide case and the people involved, they proceed to interview persons of interest. As the simulation progresses and student encounter new evidence, they will be able to submit that evidence for various types of forensic testing.

• As students proceed with their investigation, they gather evidence, test results, reports, and witness statements. These items are placed into their binder as they are created or discovered.

Deliverable: Arrest Warrant ReportObjectives: Gather enough evidence, secure

an indictment from the district attorney, and solve the case.


Setting CCJS 420


• Response Rates (CCJS 420):


CCJS 420No. of Surveys sent

No. of Responses

Response rate section

6980 27 12 44.4%

7380 19 6 31.6%

7980 30 8 26.7%

Total 76 26 34.2%

Results: Cold Case File Simulation

Results Summary: Cold Case File Simulation


This simulation exercise helped me understand the material better. 84% LARGE

It gave me a hands-on opportunity to understand the material. 88% LARGE

It deepened my understanding of the subject as it mirrored real life situations. 68% LARGE

It helped to improve my decision making skills. 75% LARGE

It improved my skills in processing, testing, and validating assumptions.

75% LARGE

It improved my skills about making connections between events and objects.

84% LARGE

It helped me visualize, conceptualize, and comprehend situations related to crime scene investigation.

92% LARGE

28


29



40%

LARGE (making connections; visualizing and

conceptualizing; engaging & approaching faculty);

MEDIUM (hands-on, mirroring, decision making,

assumption testing);

LOW (understanding, technology); LOW (negative: time

spent and age)

It improved my interaction with the instructor. 44%

LARGE (learning)

MEDIUM (technology, negative: age);

LOW (negative: time spent)

It gave me ideas for questions about the subject matter to approach the instructor. 71%

LARGE (hands-on, real-life, making connections,

visualizing, engaging),

MEDIUM (understanding, decision, assumption testing,

technology), MEDIUM (negative: age), LOW (negative: time spent)


30


Technology worked 56%

LARGE (hands-on, real-life), MEDIUM

(understanding, decision-making, assumption testing, making connections, engaging/

approaching faculty), MEDIUM (NEGATIVE: age); LOW (visualizing,

engaging w/students, no. of classes)

Time spent Average

NEGATIVELARGE (NEGATIVE: visualizing, age),

MEDIUM (NEGATIVE: hands-on, real-

life, assumption testing, making connection),

LOW (NEGATIVE: understanding,

interaction)

# of classes taken92% more than 4 LOW

Age44% between 20-30 years,

44% between 30-40 yrs

LARGE (time spent), MEDIUM

(NEGATIVE: understanding, assumption

testing, visualizing, interaction), LOW

(NEGATIVE: hands-on, decision-making,

making connections, engaging w/other students, no. of classes taken)

Gender88% female N/A


31








Setting BSBD 641


Results: Biological Attack Simulation

• Response Rates (BSBD 641):


BSBD 641section

No. of Surveys sent

No. of Responses

Response rate

9040 26 9 34.6%

9041 25 9 36.0%

Total 51 18 35.3%

Results Summary: Biological Attack Simulation


This simulation exercise helped me understand the material better.

100% LARGE (hands-on, real life, decision making, eff. Crisis

comm.., approaching faculty and technology), MEDIUM (eff. Em response, resources) LOW (consensus)

It gave me a hands-on opportunity to understand the material.

100% LARGE (understanding, real-life, crisis comm..),

MEDIUM (decision, eff. EM response, consensus,

resources)

It deepened my understanding of the subject as it mirrored real life situations.

89% LARGE (understanding, hands-on, decision-making, eff.

Crisis comm.), MEDIUM (eff. EM response, consensus,

resources)

It helped to improve my decision making skills. 89% LARGE (understanding, real-life, consensus, eff. Crisis

comm.), MEDIUM (hands-on, eff. EM response,

resources)

It improved my knowledge of establishing an effective emergency response structure during bioterrorism attacks.

100% LARGE (hands-on, real life, decision making, eff. Crisis

comm.., approaching faculty and technology), MEDIUM (eff. Em response, resources) LOW (consensus)

It improved my consensus gathering skills. 100% LARGE (consensus, resources, eff. Crisis comm.),

MEDIUM (understanding, hands-on, real-life, decision

making)

It helped me understand what resources are needed during a public health bioterrorism emergency.

94% LARGE (decision making, eff. EM response, resources, eff.

Crisis comm.) MEDIUM (hands-on, real-life), LOW

(understanding)

It helped improve my understanding of effective crisis communication.

100% LARGE (eff. Em response, consensus, eff. Crisis comm..)

MEDIUM (understanding, hands-on, real-life, decision

making)

34


35



94%

LARGE (consensus, eff Crisis comm.) MEDIUM (understanding, hands-on, decision making, eff. EM

response, resources, interaction w/faculty), LOW (real-life)

LOW (NEGATIVE: time spent, no. of classes taken)

It improved my interaction with the instructor. 50%

LARGE (decision making, consensus gathering, eff. Crisis

comm., approaching faculty) MEDIUM (understanding,

hands-on, real-life, eff. EM response, resources, engaging

with students) LOW (age), LOW (negative: no. of classes)

It gave me ideas for questions about the subject matter to approach the instructor. 71%

LARGE (understanding, real-life, decision making,

consensus gathering, eff. Crisis comm.., engaging with

faculty, technology) MEDIUM (hands-on, eff. EM

response, resources, engaging with students), LOW

(NEGATIVE: time spent) LOW (no of classes)


36


Technology worked 89%

LARGE (understanding, hands-on, real-life,

decision making, effective crisis comm.., engaging w/ students, approaching faculty)

MEDIUM (consensus, engaging with faculty,

time spent) LOW (eff EM response, resources)

LOW (NEGATIVE: age)

Time spent 67 % More than 8 hoursNEGATIVE MEDIUM (real-life,

technology), LOW ON OTHERS

# of classes taken 89% more than 4 MEDIUM (hands-on) LOW on others

Age22% 20-30 yrs, 39% 30-40 yrs,

22% 41-50, 17% 50-60 yrs

MEDIUM (negative: understanding), LOW (NEGATIVE: real-life, time spent) LOW (decision making, eff crisis comm., engaging w/faculty, no of classes)

Gender 17% female N/A


37








Results Summary

Crime Scene Cold case BioattackAGREE CORRELATIO

NAGREE CORRELATION AGREE CORRELATION

LearningUnderstanding material

84% LARGE 84% LARGE 100% LARGE (hands-on, real life,

decision making, eff. Crisis comm.., approaching faculty and

technology), MEDIUM (eff. Em

response, resources) LOW (consensus)

Hands-on experience

84% LARGE 88% LARGE 100% LARGE (understanding, real-life,

crisis comm..), MEDIUM (decision, eff. EM response, consensus, resources)

Mirroring real-life experience

84% LARGE 68% LARGE 89% LARGE (understanding, hands-

on, decision-making, eff. Crisis

comm.), MEDIUM (eff. EM

response, consensus, resources)

Improving decision making skills

82% LARGE 75% LARGE 89% LARGE (understanding, real-life,

consensus, eff. Crisis comm.),

MEDIUM (hands-on, eff. EM

response, resources)

38

Results SummaryCrime Scene Cold case Bioattack

AGREE CORRELATION

AGREE CORRELATION AGREE CORRELATION

LearningValidating and testing assumptions

80% LARGE 75% LARGE N/A N/A

Making connections between events and objects


Helps visualizing, conceptualizing


Understanding effective emergency response

N/A N/A N/A N/A 100%

LARGE (consensus, resources,

eff. Crisis comm.), MEDIUM (understanding, hands-on, real-life, decision making)

Improve consensus gathering skills N/A N/A N/A N/A 94%

LARGE (decision making, eff.

EM response, resources, eff.

Crisis comm.) MEDIUM (hands-on, real-life), LOW

(understanding)

Learn effective use of resources in EM response

N/A N/A N/A N/A 100%

LARGE (eff. Em response,

consensus, eff. Crisis comm..)

MEDIUM (understanding,

hands-on, real-life, decision making)

Improve effective crisis communication skills

N/A N/A N/A N/A 94% LARGE

39

Results Summary

40



Interaction

Student to student engagement

37%

MEDIUM

(decision making, understanding,

technology) LOW on all others

40%

LARGE (making

connections; visualizing and conceptualizing;

engaging & approaching faculty);

MEDIUM (hands-

on, mirroring, decision making, assumption

testing); LOW (understanding,

technology); LOW

(negative: time spent and age)

94%

LARGE (consensus, eff Crisis

comm.) MEDIUM (understanding, hands-on, decision making, eff. EM response, resources, interaction

w/faculty), LOW (real-life)

LOW (NEGATIVE: time

spent, no. of classes taken)

Engaging with faculty

76%

MEDIUM

44%

LARGE (learning)

MEDIUM

(technology, negative:

age); LOW (negative:

time spent)50%

LARGE (decision making,

consensus gathering, eff. Crisis comm., approaching faculty)

MEDIUM (understanding,

hands-on, real-life, eff. EM response, resources, engaging

with students) LOW (age),

LOW (negative: no. of classes)

Approaching faculty

78%

LOW - MEDIUM

71%

LARGE (hands-on,

real-life, making connections,

visualizing, engaging),

MEDIUM

(understanding, decision, assumption testing, technology),

MEDIUM (negative:

age), LOW (negative:

time spent)

71%

LARGE (understanding, real-

life, decision making, consensus gathering, eff. Crisis comm.., engaging with faculty, technology)

MEDIUM (hands-on, eff. EM

response, resources, engaging

with students), LOW

(NEGATIVE: time spent)

LOW (no of classes)

Results Summary

41



Other issuesTechnology worked

74%

MEDIUM

56%

LARGE (hands-on,

real-life), MEDIUM

(understanding, decision-making,

assumption testing, making connections,

engaging/ approaching

faculty), MEDIUM (NEGATIVE: age);

LOW (visualizing,

engaging w/students, no. of classes)

89%

LARGE (understanding, hands-

on, real-life, decision making, effective crisis comm.., engaging w/ students, approaching faculty)

MEDIUM (consensus,

engaging with faculty, time spent)

LOW (eff EM response,

resources) LOW

(NEGATIVE: age)

Time spent

Uniformly distributed

MEDIUM (approaching faculty), low otherwise

Average

NEGATIVELARGE

(NEGATIVE: visualizing, age),

MEDIUM

(NEGATIVE: hands-

on, real-life, assumption testing, making

connection), LOW

(NEGATIVE: understanding,

interaction)

67 % More than 8 hours

NEGATIVE MEDIUM (real-life, technology), LOW

ON OTHERS

# of classes taken 61% more

than 4

LOW NEGATIVE on

all, no correlation with interaction

92% more than 4

LOW

89% more than 4

MEDIUM (hands-on) LOW on others

Results Summary

42



DemographicsAge 49%

between 20-30

years, 27% between 30-40 yrs

MEDIUM (NEGATIVE: mirroring real-life; improve decision

making; assumption

testing; making connections);

LOW

(NEGATIVE: understanding, hands-on exp.,

visualizing);

LOW

(POSITIVE: engaging

w/students & faculty, no of

classes taken)

44% between 20-30 years, 44% between

30-40 yrs

LARGE (time spent),

MEDIUM

(NEGATIVE: understanding,

assumption testing, visualizing, interaction),

LOW (NEGATIVE: hands-on, decision-

making, making connections, engaging w/other students, no. of

classes taken)

22% 20-30 yrs, 39% 30-40 yrs, 22% 41-50,

17% 50-60 yrs

MEDIUM (negative:

understanding), LOW (NEGATIVE: real-life, time

spent) LOW (decision making,

eff crisis comm., engaging w/faculty, no of classes)

Gender 69% female 88% female 17% female

Conclusions

43

• Online scenario simulations add value to understanding.

• They help learning through discovery.• Provide near real-life, hands-on

experience.• Help visualize, conceptualize and make

connections between events and tools/objects.

• There is a correlation between student satisfaction and grades. Further studies with control group settings can uncover this or more facts.

• May or may not increase interaction among students. Further studies should look into this.

Conclusions

44

• There is evidence of increased student-to-faculty interaction.

• Technology glitches diminish value and satisfaction. (e.g. dead ends, loops)

• Information glitches diminish value and satisfaction.

• Age plays an important role in designing the content for the student audience.

• A negative correlation between time spent and learning the material was uncovered. Was the time spent on technical problems? Further studies should look into optimal use of time. Too much time spent may bring diminishing returns.

Conclusions

45

• The more sophisticated (game like) features make the tool more satisfactory.

• Students would like a sense of achievement at the end (e.g. solving piece of a puzzle, putting pieces together)

• Caution: Make them game-like but emphasize pedagogy.

• If designed as a group work with role playing ability may increase communication skills, help build core competencies in crisis management such as consensus gathering, effective resource allocation, understanding response structures in large scale crises and disasters.

Conclusions

46

• Recommend a move from low budget productions to sophisticated online simulation tools if budget permits.

• Expanded studies may show a link between the dollar amount spent for development and the student success outcome (Benefit-Cost Analysis).

• The tools should allow room for changing/ alternating between multiple scenarios to prevent boredom among professors and student sharing of results.

• Need buy-in from instructors. Engage them in designing the tools.

Conclusions

47

• There are successful applications with proven results about other similar simulations employed by other higher ed instutions. UMUC can emulate on that.

• The new generation as “Digital natives” learn better with game-like simulations.

Acknowledgments

48

This research was funded by the UMUC Faculty Research Grant Program.

Those who contributed to the research include Husein Abdul-Hamid, Susan Blankenship, Howard Krivan, and Howard Dobres.

Documents

Assessing the Benefits of Online Scenario Simulation Tools in Security-Related Studies at UMUC Irmak Renda-Tanali, DSc. Collegiate Associate Professor