Embed Size (px)
Citation preview
Causal Loop Diagramming for
Evaluation and Assessment
Ian Mitchell BA BSc FORS
Operational Research Unit
Analytical Decision Support
Strategy, Better Regulation and Economics (SABR)
Topics
• Introduction
• Place of Causal Loop Diagramming (CLD)
• Mechanism
• The Meeting Room example
• Experiences
• Effective Evaluations
• Weaknesses -Strengths
• Possibilities
A system is a regularly interacting or interdependent group of items forming a unified whole.
Afghanistan Sustainable Infrastructure Plan p11 2.2.7 2011
Components and
Boundaries
•Things
•Information
•Groups
Influences
•Flows
•Quantity
•Time
•Positive
•Negative
Patterns
•Feedback loops
•Reinforcing
•Balancing
•Pinch-points
•Levers
Systemigrams
Complex layered views
Quantitative models
Vensim
Informal to formal methods
Lists - Sketches - Mind maps
Causal Loop Diagramming
Post it - paper - pencil – pen
Vensim
A
B
E
C
D
F
-
+
+
+
+ +
+ A
B
E
C
F
D
Systems Diagrams
hazard management
detection
Warning And
Reporting
protection
medical
countermeasures
PRETREATMENTS
POST
TREATMENTS
casualtiescaused by nbc
attack
COLLECTIVE
PROTECTION
INDIVIDUAL
PROTECTION
DECONTAMINATION
NBC HARDENING
in protection
effectiveness ofprotection
reduced tempo due to
degradation
casualties caused by
protection
rate of nbc casualties
loss of tempo due to
hazard management
total effect on tempo
reduced combat
effectiveness
attack type
effectiveness ofmedical
countermeasures
able to detect
casualties from direct
attack
Persistent Area
Contamination
casualties from persistent
contamination
nbc threat
rate of nbc useDETERRENCE
Outcome
POINT
DETECTION
MONITORING
proportion suffering
degradation effects
<time of day>
Enemy
Manoeuvre
Capabil ity
MEDICAL
SURVEILLANCE
NBC RECCE AND
SURVEY
Own
Manoeuvre
Capabil ity
GENERIC
THERAPY
WATER TEST
effectiveness of hazard
management
confirmationof use
attack identified
PROCESS
disseminate
ENEMYDEGRADATION
EFFECT
<Enemy
Manoeuvre
Capabil ity>
OPPORTUNITIES
TO USE NBC
ENEMY NBC
CAPABILITY
HEAT STRAIN
CASUALTY
TABLE
credibi li ty ofdeterrent
AGENT
PERSISTENT
HAZARD
AGENT
PERSISTENCE
TIME
ABILITY TO DETECT
PROPORTION
IN COLPRO
TIME FOR DECON
rate own capabil i ty
reduces enemy
rate enemycapabil ity
reduces own
INITIAL ENEMY
CAPABILITY
ENEMY COMBAT
EFFECTIVENESS
OWN COMBAT
EFFECTIVENESSINITIAL OWN
CAPABILITY
CASUALTIES PER
DIRECT ATTACK
AGENT
DOWNWIND
HAZARD
MANOEUVRE
DEGRADATION
EFFECT
EFFECT OF HEAT STRESS ON TEMPO
CAPABILITY USED
TO LAUNCH ATTACK
Nbc Attacks Used
Total
Manoeuvre
Casualties
<rate of
arrival in
theatre>
<conflict begins>
information
decay time
Decision To
Use Nbc PERCEIVED UTILITY
OF NBC
<currentenemy force
ratio>
NBC ATTACKS
AVAILABLE
CASUALTIES PERPERSISTENTENCOUNTER
PERSISTENTENCOUNTER
RATE
casualties from
downwind hazards
CASUALTIES PERDOWNWINDENCOUNTER
FORWARDENCOUNTER
RATE
potential encounters
with persistent
NUMBER OF UNITSencounters with
downwind hazards
action
collect
C3IDELAY
INFORMATION
PER ENCOUNTER
detection
made
ANY
DETECTION
INFORMATION
NEEDED
GENERIC
IDENTIFICATION
agent identi fied
SPECIFIC
IDENTIFICATION
specific therapy given
<mean persistent
encounter rate>
<MEDICAL
SURVEILLANCE>
ENEMY RELEASE
OF AGENT
generic
therapy given
AREA
DETECTION
DECON
FREQUENCY
direct
contamination
WARNING
EFFECTIVENESS
ENEMY STRIKE
NBC CAPABILITY<Enemy Strike
Capability>
<own strike
effects on
manoeuvre>
<time of day>
<time in ipe
vs agent>
proportion of
time in ipe
<time in ipe due to
conventional>
Rate Persistent
Areas Lost
<personnel
multiplier>
CONFIRMATION
SPECIFIC
THERAPY
<detection
made>
<detection
made>
<AGENT
PERSISTENCE TIME>
<TIME OFDOWNWIND
HAZARD>
COLLECTION
EFFECTIVENESS
<proportionvs
manoeuvre>
PRETREATMENT
TAKE UP
PERSONNEL
LOSS RATE
<Personnel In
Theatre>
PROPORTION
WORKING AT
MAX RATE
<enemy
surrender>
<effectiveness of
locs>
<conflict
begins>
Downwind Hazards
In Forward Area
<rate forward
hazards disperse><rate forward
hazards move on>
<total
hazards>
information from
recce and survey
INFORMATION PER
RECCE AND SURVEY
<rate of nbc
use vs locs>
<rate of nbc
use vs strike>
<rate of nbc use>
<information from
recce and survey>
<total encounter
rate>
rate persistent
hazard laid
<AGENTFALLOUTHAZARD>
WARNING
FACTOR
<rate of nbc
use vs locs>
Mechanism - Objective
• Develop a systems diagram of the key factors which influence
eg caseload experienced by the Employment Tribunal System
• Use the systems diagram to:
– Understand the impact of key variables
– Explain why the system behaves in the way it does
– Identify drivers of caseload and the data describing these
– Identify potential interventions affecting drivers
Mechanism - Approach
• Build a model of the drivers and sub-drivers using a ‘systems
thinking’ approach
• Diagram will capture key variables and how they are
connected:
– The variables used are measurable
• things which can be thought of as having high or low values
• we do not actually have to measure them, for now
– The diagram captures how a change in one variable
impacts on the value of another
– We will aim to identifying and feedback loops
• virtuous and vicious circles
Mechanism – Approach
• Systems thinking is a qualitative rather than quantitative method
• There is no “right” answer
• The analysis is valid if it provides useful and valuable insights
• The technique is a tool to help you think about the problem and give you insight
• Diagrams are developed iteratively, halting when something fit for purpose has been produced
Worked Example
• The following example aims to illustrates the mechanics of buildings a systems thinking diagram
• Department X is experiencing a problem with its meeting rooms. Although a survey of meeting rooms shows that they are not well utilised the accommodation section receives many complaints that staff cannot find meeting rooms when they need them
• The Department investigates this using Systems Thinking…
– To inform its forecasts for meeting room requirements
Difficulty finding
meeting rooms
An arrow from A to B means “A causes
(or leads to) B”
(other things being equal)
A ‘+’ at the head of the arrow indicates
that an increase in the first variable leads
to an increase in the second
(other things being equal)
+ Complaints to
Accommodation
Worked Example
• The more un-booked meeting rooms there are, the easier it is (other
things being equal) to find a room for your meeting
A ‘-’ at the head of the arrow indicates that an increase in the first variable leads to a
decrease in the second (other things being equal)
- Difficulty finding
meeting rooms
+
Complaints to
Accommodation
Number of
Available meeting
rooms
Worked Example
• Because of the difficulty finding meeting rooms, staff sometimes book meeting rooms for meetings they might need to hold
• Directorates also argue for meeting rooms to be allocated for their exclusive use
• These two actions tend to reinforce the difficulty finding meeting rooms (i.e. there are two reinforcing loops)
-
R
R
Loops occur when a change in a variable leads to an impact on the value of the variable
itself. Reinforcing loops (vicious/virtuous circles) are denoted by ‘R’ on the diagram.
- Difficulty finding
meeting rooms
+ Complaints to
Accommodation
Number of
Available meeting
rooms
- +
Meeting rooms
booked “just in
case”
-
+
Meeting rooms
allocated to
specific
directorates
Worked Example Balancing loops (where an increase in a
variable tends to push the value of the
variable back down)
can be denoted by ‘B’ on the diagram. -
-
-
+
R
R
+
- Difficulty finding
meeting rooms
+
Complaints to
Accommodation
Number of
Available meeting
rooms
Meeting rooms
allocated to
specific
directorates
Meeting rooms
booked “just in
case”
• The diagram can be expanded to begin to explain how staff experience difficulty finding meeting rooms while the usage of meeting rooms is relatively low
+ +
+
+
-
+
B -
R Number of
bookings unused
Number of unused
meeting rooms
Accommodation
section perception
of sufficient
meeting rooms
Number of
meeting rooms
Approach for Workshop
Activity 1 Introduction Background and Brief overview of the method
Activity 2 Agree broad areas Consider and agree broad areas of influence that should be covered by the map, working out from Arrival of ET1s caseload
Activity 3 Generate Variables Generate key variables that affect ET caseload
Activity 4
Identify Causes, Effects, Feedbacks and Data Map how each variable affects the variable of interest and other variables in the diagram. Identify additional variables and how they impact on the system Consider the data available on these
Activity 5 Wash Up Review workshop output
CLD Experiences -2012
• BIS
– Cyber Security Sector
– Employment Tribunal Claims
– Engineers
• West Point– Cornwallis Group
– Kenya
• UK - CS Live
– Railways
• Lithuania – EURO XXV
– Economy and Population
Effective Evaluation
The number of
Effective
Evaluations
conducted
• What increases this +
• What decreases this -
• What happens after effective evaluations?
Post workshop activity
• Capture output of workshop electronically
• Communicate results and analysis to participants
• Consider implications for effective evaluations
CLD Weaknesses
• One piece of paper
• 2 Dimensions
• Number of variables
– Area needed
• Number of links
– Geometry of links
• Spaghetti
• Reflecting views rather than validating them
CLD Strengths
• Speed in use
– Immediate Engagement
• Separate policy areas become aware of
– Each other
– Their places in broader systems
– Combined effects of policies
– Gaps
• Possibilities
– Levers
– Choke-points
– Models – if useful
Cyber Security
Growth In UK
Cyber-Security
Supply Sector
Ability to
Supply
Prop. of Business
Demand going to
UK SMEs
Ability to Recruit
& Retain Skilled
Staff
Level of Relevant
Skills in UK
Awareness of Career Opps. in
C-S
Amount of Business Demand
going to UK SMEs
Foreign (esp. US)
Owned Competition
Inward Investment
in Sector
Amount of Product
Accreditation
Business
Demand
Business/Consumer
Confidence
Understanding of Risk
& Desire to take Action
Assurance
of Quality
Existence of
Kite Marks
Ability of Businesses/Consumers
to Differentiate
Consumer
Demand
Investment in
Sector
Prop. of Public Sector
Business going to UK
SMEs
Barriers to Public
Procurement
UK SME
Credibility
Access to Overseas Markets
Industry
Outreach
Amount of
Innovation
Access to
Finance
Ability to get Staff
Accredited
Marketing UK
Sector
Standard req. to get
Accreditation
Cost and Time required for
Accreditation
+
+
+
+
+
+
+
+
-
+
+
+
+
+
+
+
+
+
-
-
+
-
+
+ +
+
+
+
+
+
+
+
+
+
+ +
+ +
Employment Tribunals
Number of
ET1 Claims
SubmittedAccepted
cases
Rejected
Claims
ClaimsDropped
Permanently Claims ResolvedPrior to Full court
hearings
"Full Merits"
Court Hearing
Employment Law Claim
Options e.g
UFD,Discrimination etc.
Applicants decision
to commit
Lodging Fees
Schedule (Level 1,
Level 2, L:evel 3)
Awareness of
Law/Legal
Advice
Form designprompts - options
available in tickboxes etc.
Highest possible compensation
and likelihood (of success?) -
Expected Return?
Employer
behaviour during
conciliation
Anticipated costs forparticipants; time,
money, stress, lawyersetc.
State of
Economy
Standards of behaviour(Employers) e.g.
propensity to take actionin contravention ofEmployment Law
Company
Profitability
Conflict betweenEmployers and
Employees
Management
Skills
Earlyeducation ofpeople skills
Access to
guidance
"Natural"
litigants
Desire for
justice
Information
Receptivity?
Union
membership -
Public and Private
Knowledgeof ACAS
(emploers &employees)
ACAS helpline -referrals toconsultation(informal)
Pre Claim
Conciliation (PCC)
Access toStatutory
Redundancy Pay
Awareness of
Fees
Awareness ofcosts andawards
Drop outs
Fees for Full
Court hearings
"Stayed" cases -awaiting results of
appeals e.g. BA pilotsheld in pending pool
Awards of
costs EAT appeals
Case Law
"Shock"
coverage - media
headlines etc.
Reputation damage
(to companies or
individuals)
EC1 - newrequirement toinform ACASof intentions
Multiple claimantactions (via Unionsor solicitors etc.)
Remission scheme for fees
(including Full Court ?) e.g
claimants on universal
credit, JSA etc.
Takeup -awareness of
remissions
Settlement
Agreement
Fees "recouping
costs" - ET1
issues?
Amount of effort
required by ET to
process cases e.g
written reasons
ACAS
reputation
+
+
+
+
++
-
+
-
+
+
+
+
++
-
+
+
+
-
-
-
+
+
+
+
+
+
++
+
+
+
-
-
+
+
-
+
++
-
--
+
+
+
+ -
-
+
+
+
+
+
+
+
+
-
+
+
Underlying propensity to
litigate based on claim
demographics compared to
general population, Labour
Force Survey etc.
+
<Case Law>
<EAT appeals>
+
+
Engineers
Positive coverage of
engineering in film,
book and TV
Exhibitions (Science
Museum for example)
High Impact Real
World Events e.g.
Moon Landing,
Dambusters etc.
Reputation of Engineers &
Engineering - Building
better worlds
Successful "Products" insectors: Electronics;
Aerospace; Construction,Offshore etc in big medium
small firms
Engineers doing
engineering work
Graduate
Engineers
Engineers in
work
"Technicians"
Educators' Perception
of engineers
Number of specialist
physics teachers etc.
Bursaries to train
as Maths or
Physics teachersQuality of
teachers
Perception of
engineers by
pupils
Inter-facultycommunication e.gbetween Maths &History teachers
Pool of all potential
engineering recruits at
primary school age
Career choice
"I'm doing
engineering" 00
Engagement with primary
school pupils (age 5
onwards) eg Rolls Royce
Professional body
engagement :
IEEE, STEM
Amount and
quality of career
advice
GCSE choices
of engineering
subjects
Potential engineers
(at age 16)
Need for "A"grades fromeducational
establishment
A Level takeup in
Maths Physics etc.
Potential engineers
(at age 18)
Deprived
areas
School drop
outs
14-19 years
: UTCs??
Apprentices "jobs
with training"
200apprentices:
L-2, L-3, L-4??
16-18 funding
available
Apprenticesfrom outside
UK
18-24
funding
External to
UK - INExternal to
UK - OUT
BSc engineering
degrees
Job readiness of
graduates
Perception of
degree desirability
Current bestengineering
practices covered
Business/university
courses or liason e.g.
Jaguar Landrover
Tuitionfees/graduate debt
(price of adegree)
Costs to universityof running
engineeringcourses
+
+
+
+
+
++
+
+
+
+
+
+
+
-
-
+
-
++
+
+
+
++
+
+
+
+
+
+
+
+
+
-
-
+
+
-
+
+
+
<Exhibitions (Science
Museum for example)>
+
+
<Graduate
Engineers>
+
"League Table"
distortions
--
MSc engineering
degrees+
-
PhD
engineering
degrees
+
-
+
+
Career choice
"I'm doing
engineering" 01
+
+
Career choice
"I'm doing
engineering" 02
+
+
+
+
+
Parental
perception of
engineering
Engineers
"resting"
Professional
Institutions
-Memberships
RAEng, IOP,
IMechE,
IEEE,IOC
5.STEM
10.
Industrial
"Bursaries-
Cadetships"
1.
Aerospace
MSc funds
2. PCDL
9.Capital
Funding
4. Internships
3. Higher Level
Apprenticeships
7.
Apprentices
to SME
6.Systematzing
individual
initiatives
6.
Systematic
Campaigns
+
+
+
+
+
+
SAT Science
+
+
<Apprentices>
+
Possible analytics
• Develop qualitative descriptive models
– Consider and agree broader areas of influence that should
be covered by the map.
– Represent new policy options
• Move to quantitative descriptive model
– Identify Causes, Effects, Feedbacks and Data
• Quantify how each variable affects the variables of interest
and other variables in the diagram.
• Identify additional variables and their impacts on the system
• Consider the data available on these
• Develop a predictive quantitative model for “what if?”
• Develop a prescriptive model to suggest way ahead
Discussion
