Upload
patrick-hoffman
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security
DomainDomain
A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security
DomainDomain
Nancy J. CookePrashanth RajivanShankaranarayanan Venkatanarayanan
Arizona State University5 May 2010
Background• Education: Cognitive Psychology/Human Factors
George Mason University, B.A. New Mexico State University, M.A., Ph.D.
• Positions Rice University New Mexico State University Arizona State University & Cognitive Engineering Research Institute
• Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA
• Section Editor, Human Factors• USAF Scientific Advisory Board• National Research Council Committee on Human
Systems Integration
Relevant Research
Team Cognition Military, Cyber, and Medical Applications
Communication Analysis
Metrics for Coordination and CollaborationSponsors
• Air Force Office of Scientific Research• Air Force Research Laboratory• Office of Naval Research• Army Research Office • Leonard Wood Institute• Veteran’s Administration – MWM VERC
Cooke’s Background
Overview
• MURI and ASU Team• Team Cognition and Team Situation
Awareness• Other Team Testbeds• CyberCog – New Testbed
MURI: Computer-aided Human Centric Cyber Situation Awareness
DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office
Two fundamental limitations of Cyber Situation Awareness (C-SA)• Gap: human cognition < -- > C-SA tools
– Situation data exceeds “cognitive throughput” of human analysts• “Blind spots” in views of cyber situation for existing C-SA tools (including
auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools)
Cyber-SA Vision• Build data < -- > human decision links through innovations
– knowledge fusion– cognitive automation– artificial intelligence– visual analytics
• Awareness-driven cyber defense vs. malware behavior dependent defense • Automatic blind spot identification and monitoring techniques
MURI Partners• Professor Peng Liu, Penn State University, Overall PI • Professor Nancy Cooke, Arizona State University • Professor Coty González, Carnegie Mellon University • Professor Dave Hall, Penn State University • Professor Sushil Jajodia, George Mason University • Professor Mike McNeese, Penn State University • Professor Peng Ning, NC State University • Professor VS Subrahmanian, Univ. of Maryland • Professor John Yen, Penn State University • Professor Michael Young, NC State University
ASU MURI TeamNancy J. CookeProfessor, Cognitive Science & EngineeringCollege of Technology and Innovation
Prashanth RajivanGraduate Student Master’s in Computing StudiesCollege of Technology Innovation
Shankaranarayanan VenkatanarayananGraduate Student Master’s in Computing StudiesCollege of Technology and Innovation
Teams and Cognitive TasksTeam is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide,
perceive, design, solve problems, and act as an integrated system.
Cognitive activity at the team level= Team Cognition
Improved team cognition Improved team/system effectiveness
Heterogeneous = differing backgrounds, differing perspectives on situation
(surgery, basketball)
• Unmanned Aerial Vehicles • USS Vincennes shoots down
Iranian airbus (1988)• Challenger/Columbia accidents
tied to poor organizational decision making (1986/2003)
• Response to 9/11 reveals communication breakdowns (2001)
• Katrina response lacked coordination (2005)
• Sago Mine disaster report cites poor command-and-control (2006)
• VA Tech communications substandard (2007)
• Friendly fire incidents• Various health care mishaps
attributed to poor teamwork
Some Instances of Failures of Team
Cognition
Miracle on the Hudson
And some successes…
Response to Fargo flooding
Interactive Team Cognition in a Nutshell
Team interactions often in the form of explicit communications are the foundation of team cognition
ASSUMPTIONS
1) Team cognition is an activity; not a property or product
2) Team cognition is inextricably tied to context
3) Team cognition is best measured and studied when the team is the unit of analysis
US 2004 Olympic Basketball Team
"We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play."
A team of experts does NOT make an expert team
Collaborative skill is not additive
US 1980 Olympic Ice Hockey Team
Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey
An expert team made up of no-names…
In our UAV STE three operators must coordinate over headsets in order to
maneuver their UAV to take pictures of ground targets
Our UAV Testbed
UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded)
Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems
Payload Operator controls camera settings, takes photos, and monitors camera systems
DEMPC navigator, mission planner, plans route from target to target under constraints
Interdependence requires interaction, communication, & coordination
Three team members with inter- dependent tasks
Our MacroCog (Macro-Cognition Testbed)
MacroCog Testbed
Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations
MacroCog Roles in Current Experiment
Information Warfare Specialist
Personnel Specialist: Military
Equipment Specialist:
Land/Sea Vehicles
Personnel Specialist:
Humanitarian
Equipment Specialist: Air
Vehicles
Experimenter 1
Experimenter 2
Example of Empirical Results on Team Cognition
As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10
Mission
Team
Per
form
ance
Tm 1
Tm 2
Tm 3
Tm 4
Tm 5
Tm 6
Tm 7
Tm 8
Tm 9
Tm 10
Tm 11
• Individuals are trained to criterion prior to M1• Asymptotic team performance after 4 40-min missions (robust finding)• Knowledge changes tend to occur in early learning (M1) and stabilize• Process improves and communication becomes more standard over time
40-min missionsSpring Break
Team Situation Awareness
A team’s coordinated perception and action in response to a change in the environment
How can we exercise team SA in a testbed?
How can we measure it?
How can we intervene to improve it?
Contrary to view that all team
members need to “be on the same page”
What is Meant by Coordinated Perception and Action?
Measure of Team Situation Awareness• Change is introduced (communication breakdown, enemy in area,
storm) that will impact mission
• 2-3 team members are presented cues regarding change
• Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change
• Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly)
• Measure in terms of outcome and process – who on team was involved?
CyberCog Simulator
Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation
CyberCogSimulator – System Overview
CyberCogSimulator – Components
• Cyber Security Analyst (User)– Assigned a specific role such as Denial of Service
(Dos) specialist, Malware specialist and Phishing specialist
– Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks
– The team reaches a common consensus on the type of attack and its corresponding events
CyberCogSimulator – Components
• Master controller and Evaluator– Queries attack scenarios, events and symptoms
from the database– Distributes the events and symptoms to the
participants– Logs the interaction between participants at real
time– Evaluates and scores the participants findings
with the expected results
CyberCogSimulator – Components• Database server
– MySQL database server stores :-• Attack Scenarios• Events corresponding to attack scenarios
including some false positives & noise events• Attack Symptoms for each specialization (E.g.,
Dos, Malware , Phishing) identified• The expected results, interaction (between
participants ) logs and attack conclusion arrived at by each team for each session
User and Team Views
Legends
FunctionsFunctions
Data
CyberCog Simulator- Interaction
CyberCogSimulator- Architecture
Microsoft IIS
Database
Intra/Internet Malware Specialist
Phishing Specialist
Dos Specialist
Client Tier
Controller & View Tier
POCO’s
ADO.net
Web Services
Model Tier
Conclusion
• There are current gaps and limitations in Cyber Situation Awareness
• Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation
• CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it
Team Cognition Research Program
Testbeds: 1) UAS C22) Navy Strategic Planning
Empirical Studies in Testbed
UAS Field Data
Theory Development
ACT-R Model of Synthetic Teammate
Dynamical Systems Modeling
Measures
3540 3560 3580 3600 3620 3640 3660 3680 3700
2940
2950
2960
2970
2980
2990
Time (s)
Cum
ulat
ive
Spe
akin
g (s
)