Risk Assessment Volume 1 Setting

8/6/2019 Risk Assessment Volume 1 Setting

http://slidepdf.com/reader/full/risk-assessment-volume-1-setting 1/39



Homeland Security Institute

Melanie C. CummingsDavid C. McGarveyPeter M. Vinch

Approved by:George E. Thompson

Programs Division Manager

ANSER

Bruce W. Colletti

Homeland SecurityRisk Assessment

Volume I: Setting

June 16, 2006

RP05-024-01a

Homeland Security Institute



ii

For information about this publication or other HSI research, contact:

Homeland Security InstituteAnalytic Services, Incorporated

2900 South Quincy StreetArlington, VA 22206

Tel (703)-416-3550; Fax (703)-416-3530

www.homelandsecurity.org



iii

PREFACE

Homeland Security Institute (HSI) studies and analyses, undertaken by mutual consentbetween the Institute and the Department of Homeland Security (DHS), are organized asTasks in the annual HSI Research Plan. This report presents the results of research andanalysis conducted under

Task 24: Risk Assessment

of HSI’s Fiscal Year 2004 and 2005 Research Plans. The primary objective of Task 24 isto evaluate the applicability of both standard and emerging risk assessment methods,techniques, and tools to homeland security concerns about terrorism.

This report is intended for managers and decision makers in DHS and other governmentagencies and private sector organizations who plan, conduct, evaluate, or utilize risk assessments, but who also want to be more familiar with other basic aspects of risk

analysis. It should also prove of value to risk analysis professionals who might not haveexperience applying risk assessment to homeland security problems.

This report is the product of a collaboration led by the Homeland Security Institute (seethe Acknowledgments page). Nevertheless, the views expressed in this report are thoseof the authors. They do not necessarily represent official DHS opinion or policy.

This report supersedes all previous versions.



iv



v

ACKNOWLEDGMENTS

We thank the following HSI contributors to this report: George Murphy for providing theinitial draft of the Exercises Appendix; and Shelley Kirkpatrick for contributions to theRed Teaming Appendix. We also thank Margaret Palm for her thorough editorial review,as well as the anonymous reviewers whose comments improved this report.

Several individuals made notable contributions to this report but are no longer membersof HSI. Among them, we thank Scott Bradley, for contributions to the Decision SupportSystems Appendix; and Regan Newport, for preparing the initial drafts of the JSIVA,CARVER, and Systems Effectiveness Appendices. Other former HSI staff madesignificant contributions as well but have chosen to remain anonymous. We thank themfor their efforts, while nonetheless respecting their wishes.

Thomas Dell of the Abraxas Corporation, under contract to HSI, provided extensivecontributions to the Scenario Analysis Appendix.

The University of Virginia Center for Risk Management of Engineering Systemsprovided a report under contract to HSI that was used as the basis for the PartitionedMulti-Objective Risk Method Appendix. We thank the authors of this unpublishedreport—Yacov Haimes, Barry Horowitz, James Lambert, Erika Evans, Matthew Henry,Mark Waller, Gregory Williams, and Kenneth Crowther—for their contribution.

Innovative Decisions, Inc. made extensive contributions—particularly to the research thatinformed Volume II of this two-volume report. We thank Gregory Parnell, RobinDillon-Merrill, Robert Liebe, and Gary Smith for their efforts.

The Center for Technology & Systems Management (CTSM) of the University of Maryland contributed extensively to the research and analysis that informed thedevelopment of this two-volume report. We thank Professor Bilal Ayyub, Director of CTSM, and Dr. Mark Kaminskiy for their writings and their counsel.

Finally, we thank Robert Ross, Patrick Spahn, and Ronald Taylor of the U.S. Departmentof Homeland Security, and Dennis Buede and Michael Donnell of Innovative Decisions,Inc., for reviewing earlier versions of this report and providing valuable suggestions.

Of course, notwithstanding these valuable contributions, the authors remain solelyresponsible for the contents of this report.



vi



vii

EXECUTIVE SUMMARY

The attacks of September 11, 2001 signal that we face an intelligent and resourcefulterrorist threat. Our response must use resources wisely. Risk analysis 1 can help to

organize our thinking, guide our response, and involve stakeholders from the privatesector and federal, state, local, and tribal governments.

This two-volume report focuses on the subset of risk analysis known as risk assessment .2 However, a careful treatment of risk assessment should first acknowledge its place withina larger setting. Volume I describes that larger setting, which includes other elementssuch as risk management. 3 Moreover, the reader who is experienced in the practice of risk assessment must know how terms are being defined and concepts used in this report.Therefore, Volume I also includes an overview of the terms and concepts used in ourdiscussion of risk .4

Figure ES-1 arranges key elements in a way that suggests their inter-relationships, whilerecognizing that the details of any particular risk assessment must be tailored to thedecision or problem at hand. The figure organizes these elements into three planes, ortiers:

• The bottom tier, Mission-Based System Definition, is where risk analysisstarts. The analysis objectives and scope help identify the missions toinclude in the risk analysis; security objectives help identify systems forrisk assessment.

• The middle tier, System-Based Risk Assessment (the focus of this report),includes threat, vulnerability, and consequence assessment. Red arrowsindicate that adversaries can alter their choices, thus causing risk shifting .

• The top tier, Risk-Informed Decision Making , puts risk at the center of adecision process that considers broad objectives (e.g., financial, social,legal), constraints, and the costs and benefits of risk management options.

Volume II presents 25 primers on diverse methods, techniques, and tools of risk assessment chosen for their actual or potential use in homeland security analyses. Theseapproaches fall into two groups. Standard Approaches are used widely, while Emerging

Approaches are new to homeland security risk assessment (even if well-knownelsewhere). Volume II shows how each approach relates to the elements of systemdefinition, threat, vulnerability, consequence, and risk assessment.

1 Risk analysis is the process of assessment and management of risks.2 Risk assessment is the systematic process that evaluates the nature and magnitude of risk and its components.3 Risk management is the process that identifies, evaluates, selects, implements, and monitors actions taken to alter risk levels.4 Risk is the potential for loss or harm to systems due to the likelihood of an unwanted event and its adverseconsequences. Chapter 1 elaborates upon this definition.



viii

Figure ES-1. Homeland Security Risk Analysis Setting

Finally, Volume I includes a brief discussion of four challenges that confront homelandsecurity risk analysis and motivate the search for emerging approaches:

• Complex Systems have many constituent parts whose interactions resultin system behavior that cannot be predicted merely from a knowledge of those parts. It can be difficult to assess vulnerabilities of such systems, orthe consequences of attacking them.

• Adaptive Threats complicate efforts to assess adversary values,intentions, capabilities, and their collective impact on the likelihood of anattack.

• Uncertainty may derive from lack of relevant data, the difficulty of eliciting reliable expert opinion, and/or the propagation of individualuncertainties through a risk assessment.

• Measures and Standards refers to the problem of developingwidely-applicable schemes for characterizing, aggregating, andcommunicating the results of risk assessments. Such schemes may

include “soft” measures of consequence, for example, or improved risk visualization techniques.

In sum, the importance of risk assessment in homeland security is clear. The authorshope that this report will contribute to a greater understanding of risk assessmentprinciples, methods, techniques, and tools.



ix

CONTENTS

1. INTRODUCTION ........................................................................................ 1

1.1. BACKGROUND .................................................................................................................1 1.2. R ISK DEFINED ..................................................................................................................2

1.3. REPORT PURPOSE , SCOPE , AND STRUCTURE ...................................................................3 1.4. REFERENCES ....................................................................................................................3

2. R ISK ANALYSIS SETTING ......................................................................... 5

2.1. BACKGROUND .................................................................................................................5 2.2. TIER I: MISSION -BASED SYSTEM DEFINITION ...............................................................6 2.3. TIER II: SYSTEM -BASED RISK ASSESSMENT ..................................................................8 2.4. TIER III: RISK-INFORMED DECISION MAKING ...............................................................8 2.5. RISK COMMUNICATION ...................................................................................................9 2.6. SUMMARY .....................................................................................................................10

2.7. REFERENCES ..................................................................................................................10 3. SYSTEM -BASED R ISK ASSESSMENT ...................................................... 11

3.1. SYSTEM -BASED RISK ASSESSMENT TIER IN DETAIL ....................................................11 3.1.1. Step 1: Threat Analysis.................. .................. .................. .................. .................. ..11 3.1.2. Step 2: Vulnerability Assessment................... ................... .................. .................. ..12 3.1.3. Step 3: Consequence Assessment................... .................. .................. .................. ...13 3.1.4. Risks ............... .................. ................. .................. ................. .................. ................. .15

3.2. RISK APPROACHES AND USES OF THE RISK ANALYSIS SETTING ..................................17 3.3. SUMMARY .....................................................................................................................17

3.4. REFERENCES ..................................................................................................................21 4. C HALLENGES & E MERGING APPROACHES ........................................... 23



x



xi

FIGURES

FIGURE ES-1. HOMELAND SECURITY RISK ANALYSIS SETTING .................................................................. viii

FIGURE 2-1. HOMELAND SECURITY RISK ANALYSIS SETTING ........................................................................ 6

FIGURE 3-1. THREAT ANALYSIS PROCESS .................................................................................................... 11 FIGURE 3-2. VULNERABILITY ASSESSMENT PROCESS .................................................................................. 12 FIGURE 3-3. CONSEQUENCE ASSESSMENT PROCESS ..................................................................................... 13 FIGURE 3-4. RISK ASSESSMENT PROCESS ..................................................................................................... 15 FIGURE 3-5. NOTIONAL QUANTITATIVE DISPLAY OF RISK ........................................................................... 16 FIGURE 3-6. NOTIONAL QUALITATIVE DISPLAY OF RISK ............................................................................. 16 FIGURE 3-7. SYSTEM -BASED RISK ASSESSMENT TIER ................................................................................. 18



xii

TABLES

TABLE 3-1. STANDARD RISK ASSESSMENT APPROACHES ............................................................................ 19 TABLE 3-2. EXAMPLE USES OF THE HOMELAND SECURITY RISK ANALYSIS SETTING .................................. 20

TABLE 4-1. EMERGING APPROACHES TO RISK ASSESSMENT CHALLENGES .................................................. 24





2

during, and after events). There is also a desire to implement dual-benefit solutions (e.g.,those that address other hazards in addition to terrorism), which are important forlow-probability or low-consequence events for which the expenditure of resources mightotherwise not receive adequate priority. Finally, solutions must have limited impact onlawful activities and civil liberties.

Although much rides on reliable risk analyses, the research behind this report found nocommon unifying risk analysis framework. Although the likely explanation is that risk (as defined below) is too multi-dimensional, we also believed that:

• Risk analyses are complex analytical undertakings requiring diverseareas of expertise. Necessity forces these analyses to split intomanageable parts that must eventually be synchronized. The relationshipsamong these parts must be clearly understood.

• A non-collaborative risk analysis process that suffers unbalanced riskmanagement can yield unacceptable residual risk or risk shiftingwithout increasing security. Residual risk can occur when

communication fails between decision making levels, such as when onelevel of government acts with the expectation that other levels will addressspecified risks. Without a commonly understood risk analysis process,this expectation may be in vain. When unbalanced risk managementyields risk mitigation actions in one area and few actions in another,adversaries may shift focus to the exposed areas.

We now turn to the necessary starting point for this report: the definition of risk.

1.2. Risk DefinedRisk analysis involves persons from diverse backgrounds who may not share a commondefinition of risk. Indeed, risk means different things to different people, and this ispartly due to how each community perceives and discusses risk. We propose thedefinition below, one that emerged from our reviews of over fifty risk assessmentmethods, techniques, and tools chosen for their use or promise to homeland security, andof over thirty risk assessment frameworks from government, academia, and industry:

Risk is the potential for loss or harm to systems due to thelikelihood of an unwanted event and its adverse consequences.

Potential implies uncertainty, 1 which is inherent in the likelihood of the unwanted eventand in the nature and severity of its adverse consequences. Loss or harm includes allnegative consequences, tangible or not. A system is a set of elements (people, property,environment, and processes) that act together in a coordinated manner to further specificfunctions, represented by outputs. The unwanted event is an occurrence that triggersadverse consequences , and likelihood refers to both the occurrence of the event and its

1 There are two types of uncertainty. Aleatoric uncertainty is rooted in randomness, such as flipping a coin. Epistemicuncertainty is rooted in lack of knowledge or cognition, such as where an attack will occur. Risk assessment and risk management consider uncertainties in physical, economic, political, and sociological dimensions of a system’sbehavior.



3

potential adverse consequences. Although often used interchangeably, we treat probability as a quantitative measure of likelihood .

1.3. Report Purpose, Scope, and Structure

The two-fold purpose of this two-volume report is to present short primers on diverse risk assessment methods, techniques, and tools (Volume II), and give a non-technical generalsketch of risk concepts and terms that surround such approaches (Volume I).

This report addresses risk assessment before a terrorist attack occurs. 2 Although thisreport seems applicable to other hazards, we do not explore this because it is outside ourscope. Notably absent from the Volume II appendices is the risk analysis methodologyfor critical infrastructure and key asset protection that ASME (formerly the AmericanSociety of Mechanical Engineers) is developing for DHS. This is omitted because itsreport has not yet been released [Hutchinson, 2005].

Chapter 2 surveys a three-tiered homeland security risk analysis setting and brieflydiscusses the two types of risk communication. Chapter 3 describes the risk assessmenttier and identifies some standard approaches to risk assessment. Chapter 4 summarizesthe challenges in conducting homeland security risk assessments and describes promisingemerging approaches that address these challenges.

1.4. ReferencesHaimes, Y. (2004). Risk Modeling, Assessment, and Management . Hoboken, NJ: JohnWiley & Sons, Inc.

Hutchinson, H. (2005, January). Calculating Risks. ASME Mechanical Engineering Magazine . Retrieved July 2005 from

www.memagazine.org/backissues/jan05/features/calcrisk/calcrisk.html .Society for Risk Analysis (SRA). (2005). Glossary of Risk Analysis Terms . RetrievedNovember 7, 2005, from http://sra.org/resources_glossary.php .

2 Since there is no universally accepted definition of terrorism, we avoid debating such definitions.



4



5

2. RISK ANALYSIS SETTING

“If properly applied, threat and risk assessments can provide ananalytically sound basis for building programmatic responses to various

identified threats, including terrorism.” —U.S. General Accounting Office, Combating Terrorism: Threat and Risk Assessments Can Help Prioritize and Target Program Investments ,GAO/NSIAD-98-74, April 1998

2.1. BackgroundIn building this report whose focus is on risk assessment methods, techniques, and toolschosen for their use or promise to homeland security, we reviewed over fifty suchapproaches. However, since we also wanted to give a sense of risk assessment’s role in alarger setting, we also reviewed over thirty risk analysis frameworks from government,

academia, and industry. In particular, we valued frameworks that were:

• Logical , so that stakeholders and risk analysts can grasp concepts using acommon vocabulary

• Comprehensive , so that no steps are missing• Flexible , so that the framework can adapt to changing threats, cut across

homeland security application areas, and support examination of emergingthreats

• Decision-Focused , so that the framework can support risk management• Homeland Security-Focused , so that the unique challenges of homeland

security, such as responding to adaptive adversaries, can be met

Although no risk framework met all the above criteria, we found these widely sharedcharacteristics:

• A phase for defining objectives and system boundaries. As describedbelow, this corresponds to Tier I, “Mission-Based System Definition,” of the three-tiered risk analysis setting shown in Figure 2-1.

• Phases that addressed system vulnerabilities and consequences arisingfrom faults in (or threats to) system components. (Some frameworksconsidered adversaries in place of faults.) Collectively, these phases

correspond to Tier II, “System-Based Risk Assessment.” • A phase that considered how to reduce the likelihood of an unwanted

event, reduce vulnerabilities, or reduce potential adverse consequences.This corresponds to Tier III, “Risk-Informed Decision Making.”

These criteria and common characteristics helped us to craft a perspective of risk assessment’s role in a larger setting, a perspective that can help illuminate the value of each method, technique, or tool described in Volume II. In turn, Figure 2-1 evolved as a



6

homeland security risk analysis setting 1 that sketches this perspective’s landscape, pointsout landmarks, and identifies risk assessment’s notional place without prescribing theconduct of any particular risk assessment (a matter outside this report’s scope).

In this chapter we describe each tier of this risk analysis setting, ending with a brief discussion about risk communication, which cuts across all tiers. Chapter 3 expandsdiscussion of the middle tier.

Figure 2-1. Homeland Security Risk Analysis Setting

2.2. Tier I: Mission-Based System DefinitionA risk analysis begins with objectives and scope of the study (these are the lower tierinputs). Example objectives might be to assess the risk: of a specific terrorist event; to amission, system, asset, or intangible value; or from diverse threats. The scope of a risk analysis can be defined via suitable questions. For instance, which threats andconsequences will be considered, and what are their time frames? What geographicallimits and populations shall be considered? Which attributes of society, outputs, andsystems will be included, and what types of consequences – physical, health, economic,

political, social, psychological, environmental – will be considered? What is theappropriate level of analytic resolution?

The objectives and scope are used to identify the missions to study, i.e., theresponsibilities that define the essential purpose of organizations and enterprises. Forexample, DHS missions include: secure the American homeland; protect the Americanpeople; and prevent and deter terrorist attacks [DHS, 2004]. Other missions can include

1 Red arrows indicate the influence of intelligent adversaries who can alter their capabilities or targets at will.



7

those of social systems (e.g., the Constitution gives missions of the U.S. government) andof organizations embedded in infrastructures (e.g., the mission of the U.S. air trafficcontrol system is to ensure safe and efficient air transport). Each mission has securityobjectives, e.g., prevent attacks, protect assets and infrastructure, provide warning,mitigate damage, and recover from attacks.

Next, missions and their security objectives are used to identify the systems that reside inthe lower tier. These systems have the following characteristics:

• Boundaries, Inputs, and Outputs. Systems exchange inputs and outputsacross boundaries that separate one system from another.

• Metasystems, Subsystems, and Components. A complete description of a system includes both its own elements—the subsystems and componentsthat compose it—and its relationships to the larger systems of which ititself is an element. In particular, homeland security analysts may want toconceptualize the various national, regional, or local infrastructures assystems of systems where each system is composed of subsystems and ispart of a larger system (metasystem). Subsystems that are particularlyrelevant to risk analysis are security subsystems (that reducevulnerabilities) and consequence management subsystems (that reduceconsequences of an attack upon the system).

• Critical Assets. The system definition will often identify assets that arecritical to the operation of the system. Critical assets are identified on thebasis of the potential consequences of a successful attack by an adversaryrather than on the probability that the attack will be successful [GAO,1998].

• Interdependencies exist across system boundaries and may be subject to

disruptions that would echo widely. Some types of interdependencies arephysical (flow of materials), information technology (communicationflow), geographical (collocation), and logical (such as infrastructureslinked through financial markets).

Systems thus identified in Tier I have high value to homeland security and in turn, movethe risk analysis process to Tier II, which is system-based risk assessment.



8

2.3. Tier II: System-Based Risk AssessmentThe middle tier in the risk analysis setting addresses the steps in system-based risk assessment discussed in Chapter 3. Risk assessment involves these three factors:

• Threats encompass the capabilities and intentions of an adversary toundertake actions that can harm us. In a risk assessment, threat ismeasured by the likelihood of a specific attempted attack, a measure thatmay be exceedingly difficult to obtain.

• Vulnerabilities are system attributes that an adversary can exploit [GAO,1998]. Systems can be vulnerable to an attack or to propagated damagefrom an attack. Such weaknesses can occur in design, implementation, oroperational practices. A system’s vulnerability to attack can be measuredby the likelihood of a successful attack.

•

Consequences are the outcomes or effects of an attack, generallyestimated as the expected range of loss or damage from a successful attack [GAO, 1998]. Consequence assessments consider immediate, short-term,and long-term effects; proximate and distal effects; direct and indirecteffects; and inherent capacity and resilience of affected systems. Thesemay all be provided by a vulnerability assessment of the system, or in aconsequence assessment.

The above factors address the following questions that define the process of risk assessment for homeland security applications [Kaplan and Garrick, 1981]:

• What can happen?• What are the consequences?• What are the likelihoods?

With these questions addressed by the steps found in Tier II, the risk analysis processadvances to Tier III. This is where risk-informed decision making occurs.

2.4. Tier III: Risk-Informed Decision MakingMiddle tier outputs flow to the top tier in which risk-informed decision making (risk management) occurs. Its participants are policy makers and senior executives whowrestle with social values and ethics, and with concerns of federal, state, and local

stakeholders and constituents. In this most difficult of tiers, risk assessment resultscombine with constraints, conflicting objectives, and risk mitigation alternatives thatcharacterize the complex decision making under uncertainty that is risk management.

As noted earlier, this report focuses upon the middle tier, and so discussion of the top tieris beyond our scope. We simply say that sound risk analysis enables the timely andadequate execution of risk management whose central question is “What should bedone?” The ensuing answers aim at reducing threats, reducing vulnerabilities, andmanaging consequences. However, if these answers are to be deemed credible and



9

acceptable by all stakeholders in the risk analysis process, then its many “moving parts”must have adequately communicated their contributions to other process activities. Thisis no small feat and is enabled when care is given to sound and timely risk communication, to which we now turn.

2.5. Risk CommunicationAs noted earlier, risk communication cuts across all tiers because it pervades risk analysis. It is an interactive process in which stakeholders exchange risk-relatedinformation for the purpose of making informed decisions [National Research Council,1989]. Stakeholders include the public, health care providers, first responders, publicaffairs officials, security professionals, asset owners, subject matter experts, risk analysts,and policy and decision makers. Each perceives risk differently, each has different risk tolerance levels, each ponders and discusses risk according to the customs and idioms of their community, and each seeks a common understanding of collective concerns [PublicHealth Service, 1995]. Risk communication addresses all these matters.

There are two types of risk communication:• Public Risk Communication provides information used to make

judgments about risks to health, safety, and environment [Morgan et al.,2004]. It combats fear and uncertainty by educating the public whoseperceptions of risk are affected by many factors [Ropeik and Slovic,2003]. Unfortunately, a terrorist attack raises this distress because theterrorist’s thinking may be unknown, unknown incidents may yet follow,or risk assessments of catastrophic events may be useless [Slovic, 2003].Worse yet, terrorism triggers dark emotions fueled by strangecircumstances, intense dread, involuntary exposure, lack of control,catastrophic consequences, and intentional malice [Slovic, 2003].Furthermore, when public information becomes known to the terroristsand to their supporters, matters can grow worse. Nevertheless, risk communication helps people to handle fear by educating them on actionsto take [Gray, 2003].

• Internal Risk Communication helps risk analysts, managers, anddecision makers achieve a shared grasp of risks assessments, managementdecisions, and vital information. It requires stakeholders to understandtheir different perspectives; think strategically about who is involved andat what point; actively listen to diverse viewpoints; and adjust thinkingbased on feedback and evaluation. This communication is made difficult

by the diverse ways that each stakeholder community views, measures,and computes risk, and how they see the situation. Decision makers whocannot become involved in detailed risk assessment need clear guidanceon how to use its results [NRC, 2004].



10

2.6. SummaryThis chapter presents a three-tiered homeland security risk analysis setting and brieflydescribes its tiers of Mission-Based System Definition, System-Based Risk Assessment,and Risk-Informed Decision Making. Because this report’s focus is upon risk assessment, discussion of Tiers I and III suffices to paint an idea of the larger settingwithin which risk assessment resides. Because risk communication is present in all tiersand thus is cross-cutting (being the glue that holds a risk analysis together), its discussionstood alone.

2.7. ReferencesGray, G. (2003, February). Organizing to Confront Terrorism: The Role of Risk Communication . Presented at 2003 National Health Policy Conference. Washington, DC.Retrieved August 3, 2005, from www.academyhealth.org/nhpc/2003/gray.pdf .

Kapan, S. and Garrick, B.J. (1981). On the quantitative definition of risk. Risk Analysis,

1(1):11-27.Morgan, M.G., Fischhoff, B., Bostrom, A., and Atman, C. (2002). Risk Communication:

A Mental Models Approach . Cambridge, UK: Cambridge University Press.

National Research Council. (1989). Improving Risk Communication . Washington, DC:National Academies Press. Retrieved August 3, 2005, fromhttp://books.nap.edu/books/0309039436/html/index.html .

Ropeik, D. and Slovic, P. (2003, June). Risk Communication: A Neglected Tool inProtecting Public Health. Risk in Perspective , 11(2). Retrieved August 3, 2005, fromwww.hcra.harvard.edu/pdf/June2003.pdf .

Slovic, P. (2003, October 27). A Difficult Balance: Risk Communication in an Age of Terrorism . Presented at 2003 Institute of Medicine Annual Meeting. Washington, DC.Retrieved August 2, 2005, from www.iom.edu/Object.File/Master/16/283/0.pdf .

U.S. Department of Homeland Security (DHS). (2004). Securing Our Homeland: U.S. Department of Homeland Security Strategic Plan . Washington, DC.

U.S. Government Accounting Office (GAO). (1998). Combating Terrorism: Threat and Risk Assessments Can Help Prioritize and Target Program Investments . GAO/NSIAD-98-74. Washington, DC.

U.S. Nuclear Regulatory Commission (NRC). (2004, December). Effective Risk Communication: Guidelines for Internal Risk Communication . NUREG/BR-0318.

U.S. Public Health Service. (1995, February/March). Risk Communication: Workingwith Individuals and Communities to Weigh the Odds. Prevention Report . RetrievedAugust 3, 2005, from http://odphp.osophs.dhhs.gov/pubs/prevrpt/Archives/95fm1.htm .



11

3. SYSTEM-BASED RISK ASSESSMENT

"Our analysis of the threats and risks will drive the structure, operations, policies, and missions of the Department, and not the other way around.

We will not look at the threats and our mission through the prisms of the Department's existing structures and functions. Instead, we will analyzethe threats and define our mission holistically and exhaustively, then seek to adapt the Department to meet those threats and execute that mission."

—Michael Chertoff, Secretary of the Department of Homeland Security.Testimony before the U.S. House Appropriations Homeland SecuritySubcommittee on the President's Fiscal Year 2006 budget, March 2, 2005

3.1. System-Based Risk Assessment Tier in DetailThis chapter discusses the three iterative steps in a system-based risk assessment (threat

analysis, vulnerability assessment, consequence assessment), the communication, display,and determination of risk, and example applications tied to the risk analysis settingdescribed in Chapter 2. At the end of this chapter, Figure 3-7 gathers Figure 3-1 throughFigure 3-4 into a summary display.

3.1.1. Step 1: Threat AnalysisThreat analysis gathers and analyzes intelligence and information about adversaries, andconcludes with the assessment of an attack’s likelihood. The components of threatanalysis appear in Figure 3-1 (which magnifies the middle tier’s Threats node) whose redarrows show how vulnerabilities and consequences influence threat analysis. The redarrows also indicate that intelligent adversaries will alter their capabilities or targets if theycannot achieve their aims. A complete threat analysis describes scenarios that lead tosuccessful attacks, adversary capabilities and intentions, and likelihood of attack.

Methods and techniques of threat analysis include Event, Probability, and Decision Trees(Appendix E), Fault, Success, and Attack Trees (Appendix I) and, when treating rareevents, the Partitioned Multi-Objective Risk Method (Appendix N). Scenario Analysis(Appendix P), the Analytic Hierarchy Process (Appendix A), or Expert-Opinion Elicitation(Appendix G) can be used to surmise missing probabilistic data on threats.

Figure 3-1. Threat Analysis Process



12

3.1.2. Step 2: Vulnerability Assessment

Vulnerability assessment (VA) examines the ability of a system to withstand attack. VAsassess the likelihood of success of a staged attack, identify exploitable weaknesses insystems, and estimate the effectiveness of protective security measures [GAO, 1998]. Key

VA elements appear in Figure 3-2 (which magnifies the middle tier’s Vulnerabilities node)in which security system capabilities include guards, gates, and locks for a physicalsystem; firewalls and virus scans for an IT system; or biomaterials access controls,biosurveillance, and filters for biosystems. Although overlapping security systems mayprovide extra protection, they need not eliminate vulnerabilities. For instance, flaws in theairline industry’s security system were exploited on 9/11, allowing an airborne attack thatbypassed security systems at the World Trade Center.

Figure 3-2. Vulnerability Assessment Process

A VA has three steps:• Evaluate security system components to determine their individual

likelihoods of defeat by different types of attacks.• Identify how an attacker could compromise each critical asset to bring about

system failure.• Identify pathways by which an adversary could compromise each critical

asset.

There are four categories of VA:• Checklists/Questionnaires provide a qualitative evaluation of protective

subsystems. Although no formal analysis of threats or pathways is done,components may be found that need strengthening or improvement.Relevant methods and techniques include Failure Modes and EffectsAnalysis (Appendix H) and the Joint Staff Integrated VulnerabilityAssessment (Appendix L).

• Rating/Scoring makes criteria-based non-probabilistic quantitativeevaluations of components and systems. Scores combine into an overallrating for the component or system, as in CARVER (Appendix C).

• Testing determines the vulnerability of components or systems bysubjecting them to simulated attacks. Exercises (Appendix F) are a way toconduct testing.



13

• Modeling (usually mathematical) simulates or characterizes a system andits vulnerability to attack. Relevant approaches include Event, Probability,and Decision Trees (Appendix E), Fault, Success, and Attack Trees(Appendix I), Monte Carlo Simulation (Appendix M), and SystemEffectiveness Assessment (Appendix Q).

3.1.3. Step 3: Consequence AssessmentWhen adversaries successfully exploit system vulnerabilities, consequences ensue thataffect physical and mental health, the economy and environment, society and politics, andnational security. Figure 3-3 (which magnifies the middle tier’s Potential Consequencesnode) shows the following concerns of consequence assessment:

• Direct effects upon system components• Indirect effects upon society, infrastructure, the economy, and other systems• Capabilities of consequence management systems (defined in the lower

tier’s systems) to mitigate adverse effects

Figure 3-3. Consequence Assessment Process

Some rules of thumb emerge from data that describes actual terrorism-relatedconsequences. For instance, consequences of the 9/11 and Tokyo subway attacks led theHomeland Security Council Planning Scenarios to anticipate a 10:1 ratio of uninjured:injured seeking medical attention [HSC, 2004]. Indirect economic effects werealso suggested: of those businesses that close after a moderate disaster, at least 43% neverreopen, and 29% cease within two years of reopening [HSC, 2004]. Consequences mayalso be estimated using historic data from relevant non-terrorism incidents, e.g., HAZMATspills. When data cannot be found, subject matter experts can be used, noting that theirbiases and the complexity of consequences require a structured elicitation process thatyields sound information (Appendix G).

The assessment of direct consequences often relies upon threat-dependent models, such asthose below, to shed light where data and experts cannot. Results often require translationinto terms meaningful to a decision maker, e.g., how radiation dosage equates to deaths.

• Chem-Bio Models forecast physiological effects of a chemical orbiological agent released in a population.

• Nuclear Models forecast physical consequences of a nuclear explosion.



14

• Explosion Models forecast physical consequences from conventionalexplosions.

• Fire Models forecast consequences from incendiary events.• Electromagnetic Response Models forecast effects from electromagnetic

(microwave) radiation.• Cyber Attack Models forecast disruptions to cyber infrastructure/security.• Transport Models forecast how an agent propagates through the

environment. Two such types of model are fluid dynamic and hydrographicmodels (air-water propagation) and network flow models (movementbetween system entities, e.g., transmitted information, poison through foodproduction, propagation of contagion).

Other types of models assess the indirect consequences upon the economy andenvironment, society, infrastructure, and other systems. Such models propagate effectsover time, space, psychological, and social dimensions.

• Economic Models forecast effects such as those to the economy resultingfrom the 9/11 grounding of commercial airlines, and often overlap withinfrastructure models (see Appendix K Input-Output Modeling).

• Health and Environmental Models forecast health and environmentaleffects from, for example, a nuclear, biological, or chemical event thatcascades through an area or population. These may also use fluid dynamicand hydrographic models, or social network (Appendix Y) and populationmobility models that track the spread of contagion.

• Infrastructure Models forecast the effects inflicted upon infrastructureaffected by damaged elements. Some models address interdependencies

among infrastructures, such as that between electric power andtelecommunications (the former uses the latter for control, while the latterneeds power to operate). This class of models includes the DHS CriticalInfrastructure Protection Decision Support System (CIP/DSS, Appendix D)[Bush, Deland, and Samsa, 2004], the University of Virginia’s InoperabilityInput-Output Models (Appendix K) [Haimes et al., 2005], and the NationalInfrastructure Simulation and Analysis Center (NISAC) models (AppendixR) [Wimbish and Sterling, 2003].

• Sociological, Political, and Psychological Models forecast the political,psychological, and sociological effects of an event upon individuals orsociety.

Other Volume II appendices that also address indirect consequences are BayesianNetworks (Appendix B), Decision Support Systems (Appendix D), Influence Diagrams(Appendix J), and Monte Carlo Simulation (Appendix M).



15

3.1.4. RisksWe have just sketched the three middle tier steps in system-based risk assessment, and nowaddress the determination, display, and communication of risk that comprise the “Risks”node of Figure 3-4.

Risk Determination. This is based upon:• Likelihood of an attempted attack (Threats node)• Likelihood that the attempted attack is successful (Vulnerabilities node)• Consequences of a successful attack (Potential Consequences node)

Figure 3-4. Risk Assessment Process

Calculations are both qualitative and quantitative (for instance, the former may considerdescriptive categories of each of the above three risk factors). Quantitative analysis willoften treat likelihoods as probabilities that when multiplied (presuming their underlyingevents are independent) yield the probability of a successful attack:

Probability of successful attack =

(Probability of attempted attack) *

(Probability of successful attack, given attempted attack)

It is tempting to combine the probability of successful attack with some measure of potential consequences, in an attempt to produce a single measure of risk (e.g., expectedadverse consequences). Such operations can be problematic, for two reasons. First, it ispossible to thereby lose information that may be important to a decisionmaker. Forexample, a simple expected-value computation implicitly assumes that a decisionmaker’spreferences are symmetric with respect to probability and consequence (i.e., that ahigh-probability low-consequence event is equivalent to a low-probabilityhigh-consequence event)—and this may not be the case. Second, the potentialconsequences may themselves be multi-dimensional.

Risk Display. The display of risk includes graphs, tables, and probability distributions inthe form of cumulative probability or exceedance distributions [Ayyub, 2003]. The choiceof display depends on the type of analysis (qualitative or quantitative) and stakeholderpreferences, and here we depict simple displays.



16

Two elements of risk (probability and consequence) suggest the 2-dimensional display of risk found in Figure 3-5. Each element has its own axis, and uncertainties in each reflectas line segments that form a cross to depict the range of risk. That is, the event’slikelihood falls between values a and b, and adverse consequences fall between values c and d .

Figure 3-5 . Notional Quantitative Display of Risk

Figure 3-6 uses a table ( risk matrix ) to qualitatively measure likelihood and consequences.Cell entries identify the degree of risk (low, medium, high) represented by the likelihood-consequence pairs.

Figure 3-6. Notional Qualitative Display of Risk

Internal Risk Communication. Communication of risk from analysts to decision makersis the ultimate purpose of the middle tier (whose outputs feed the risk management effortsof the top tier). This bears careful attention because stakeholders come from diverse levelsof government and industry, have diverse information needs, and grasp and discuss risk according to their community’s customs and idioms. Effective internal risk



17

communication requires a common vocabulary that all understand, especially those wholack a background in risk.

In general, internal and public risk communication is integral to the entire risk process. Ithelps achieve an information exchange among decision makers and stakeholders (whichincludes the public) while dealing with differences in risk perception and risk tolerance. Inaddition to the overall assessment of risks, this exchange includes missions involved,security objectives, mission-critical systems, critical system assets, key interdependencies,adversary threats, critical exploitable vulnerabilities, and consequences of a successfulattack.

3.2. Risk Approaches and Uses of the Risk Analysis SettingRisk assessment has many methods, techniques, and tools (collectively called approaches):a method is a set of techniques; a technique is a set of procedures; and a tool is a decisionor computational aid (e.g., software) that implements techniques or methods. Table 3-1presents several standard approaches (found in Volume II), where a marking indicatesapplicability to system definition, threat analysis, vulnerability assessment, consequenceassessment, and risk assessment.

Table 3-2 applies the risk analysis setting to notional homeland security challenges. Thetable displays a high-level summary of Tiers I-II steps applied to four hypothetical risk analyses. The table lists an objective and scope to guide the analysis, and then gives theelements of mission-based system definition: missions, security objectives, and systemsrelevant to the risk analysis.

3.3. SummaryThe System-Based Risk Assessment middle tier assesses threats, vulnerabilities,consequences, and their relationships associated with the system found in the lower tier of the homeland security risk analysis setting. Threat analysis describes scenarios that lead tosuccessful attacks, adversary capabilities and intentions, and likelihood of attack. Whilevulnerability assessment examines the ability of a system to withstand attack, consequenceassessment addresses the physical, mental, economic, environmental, political, andsecurity effects that arise when vulnerabilities are exploited. The fourth process in themiddle tier addresses the quantitative and qualitative determination of risk, how toeffectively display risk, and the internal communication of risk from analysts to decisionmakers. Middle tier results support the risk-informed decision making (risk management)that occurs in the upper tier.



18

Figure 3-7. System-Based Risk Assessment Tier



19

Table 3-1. Standard Risk Assessment Approaches

Name

S y s

t e m

D e f i n

i t i o n

T h r e a

t

A s s e s s m e n

t

V u l n e r a

b i l i t y

A s s e s s m e n

t

C o n s e q u e n c e

A s s e s s m e n

t

R i s k

A s s e s s m e n

t

A p p e n

d i x

Analytic Hierarchy Process X R X X A

Bayesian Networks X M X X B

CARVER X R X C

Decision Support Systems X X X D

Event Trees, Probability Trees, andDecision Trees X M X X E

Exercises X T X X F

Expert-Opinion Elicitation X C X X GFailure Mode and Effect Analysis X C X X H

Fault Trees, Success Trees, and AttackTrees X M X X I

Influence Diagrams X M X X J

Input-Output Modeling X K

Joint Staff Integrated VulnerabilityAssessment C L

Monte Carlo Simulation X M X X M

Partitioned Multi-Objective Risk Method X X X N

Probabilistic Risk Assessment X M X X O

Scenario Analysis X X C X X P

System Effectiveness Assessment M Q

Table 3-1 Legend

The notation below is used in the Vulnerability Assessment column, consistent with the four types ofvulnerability assessment approaches in §3.1.2:• (C) Checklists/Questionnaires • (R) Rating/Scoring • (T) Testing • (M) Modeling





21

3.4. ReferencesAyyub, B.M. (2003). Risk Analysis in Engineering and Economics . London: Chapman

and Hall/CRC.Bush, B., Deland, S., Samsa, M. (2004, April 15). Critical Infrastructure Protection

Decision Support System (CIP/DSS) Project Overview . LA-UR-04-5319. Los AlamosNational Laboratory. Retrieved November 2005 fromhttp://public.lanl.gov/bwb/do/c3deaa7498e3cda534456f844c69c4d6.pdf .

Haimes, Y.Y., Horowitz, B.M., Lambert, J.H., Santos, J.R., Lian, C., and Crowther, K.G.(2005). Inoperability Input-Output Model for Interdependent Infrastructure Sectors. I:Theory and Methodology. ASCE Journal of Infrastructure Systems, 11(2):67-79.

Homeland Security Council (HSC). (2004, July). Planning Scenarios ExecutiveSummaries. Washington, DC.

U.S. Government Accounting Office (GAO). (1998). Combating Terrorism: Threat and Risk Assessments Can Help Prioritize and Target Program Investments. GAO/NSIAD-98-74. Washington, DC.

Wimbish, W. and Sterling, J. (2003, August). The National Infrastructure Simulation and Analysis Center (NISAC): A New Contributor to Strategic Leader Education and Formulation of Critical Infrastructure Policies and Decisions . Center for StrategicLeadership Issue Paper, Volume 06-03. U.S. Army War College. Retrieved November2005 from www.lanl.gov/source/orgs/d/nisac/pdfFiles/nisac.pdf .



22



23

4. CHALLENGES & EMERGING APPROACHES

The Secretary of the Department of Homeland Security aims at risk-informed decision-making within the homeland security community. This aim encompasses a full spectrum

of homeland security issues that includes policy formulation, resource allocation, andoperations. Defensible and sound decisions in this area can be made possible by risk analyses that are scientifically credible and transparent in their assumptions, data sources,and treatment of uncertainty.

A step in this direction is the creation of a “tool box” of sound, useful, and adaptable risk analysis methods, techniques, and tools, such as the standard approaches found in Table3-1. Other approaches are needed for the following challenges:

• Complex Systems found in homeland security may resist the conventionalapproach of specification by decomposition. Their complexity is rooted inthe sheer number of interconnected dynamic components, confoundinginterdependencies with national infrastructure, and “persons in the loop.”That is, the presence of people in a homeland security system introducescomplexities associated with social systems, markets, and a host of otherconcerns. Complex systems can exhibit self-organization, emergentbehavior, and adaptation to the environment.

• Adaptive Threats speak to an intelligent, resourceful, and adaptiveterrorist threat that constantly evolves (perhaps forcing us into responsesthat unduly rely upon speculation), won’t submit to defeat, and is inharmony with social and cultural influences that we cannot or will notunderstand adequately.

• Uncertainty is rooted in random behavior, lack of knowledge, orignorance (§1.2 defines aleatoric and epistemic uncertainty). Whileepistemic uncertainty of threats compromises risk assessments, largealeatoric uncertainty may cripple vulnerability and consequenceassessments. Risk analyses are afflicted by both types of uncertaintybecause these are present in human behavior and in the physical,economic, political, and sociological dimensions of system behavior.

• Measures and Standards found in homeland security risk assessmentscan be ill-defined (or meaningless to some) because there is no standardway to stage and interpret results now created by diverse communitieswho trust in their own approaches. This condition will persist wheneverlocal analyses of diverse infrastructures inherently conflict with “bigpicture” analyses that strive to be holistic. The pressing need for “soft”psycho-social measures will upset this condition even more.

The following “emerging” analytic techniques show promise addressing the abovechallenges (Table 4-1). An emerging technique is one that is new to homeland securityrisk assessment, even if it is well-known and used in other disciplines.



24

• Agent-Based Simulation (Appendix R) models complex systems viacomputer-generated “agents” interacting in a virtual environment in whichthey sense, learn, act, and communicate. Agent-based simulations havesupported analyses of epidemics, evacuations, and economic systems.

• Computer-Enhanced Scenario Analysis (Appendix S) develops and

analyzes uncertain futures using computer-driven scenario simulations.• Game Theory (Appendix T) studies opposing “players” strategies for

achieving an optimal solution.• Multi-Objective Decision Analysis (Appendix U) is a decision analysis

method that compares alternatives under conflicting objectives. Thistechnique helps evaluate the value that an adversary places upon a target.

• Precursor Event Analysis (Appendix V) studies those operationalelements that constitute the important accident sequences that lead toaccidents in complex systems.

• Prediction Markets (Appendix W) use small markets that trade contractson uncertain events. By studying market activity, analysts may gleaninsight into the likelihood of specified threats.

• Red Teaming (Appendix X) aims at understanding the adversary’sperspective in order to identify one’s own vulnerabilities and to challengeone’s own assumptions regarding threat intentions and capabilities.

• Social Network and Dynamic Network Analysis (Appendix Y) modelssocial relationships among entities in a network. Social network analysishas been used to determine likely threats, and dynamic network analysishas contributed to consequence assessment within sociological systems.

Table 4-1. Emerging Approaches to Risk Assessment Challenges

Homeland Security RiskAssessment Challenge

Approach

C o m p

l e x

S y s

t e m s

A d a p

t i v e

T h r e a

t s

U n c e r t a

i n t y

M e a s u r e s a n

d

S t a n

d a r d s

Agent-Based Simulation X X X

Computer-Enhanced Scenario Analysis X X

Game Theory X XMulti-Objective Decision Analysis X X X

Precursor Event Analysis X X

Prediction Markets X X

Red Teaming X X

Social Network and Dynamic NetworkAnalysis X X X



25

SummaryHomeland security risk assessment requirements challenge our capabilities to assess risksto all that we value. Throughout the iterative processes of risk assessment and risk management, there are opportunities to use existing methods, techniques, and tools, whileemerging techniques hold promise addressing difficult aspects of homeland security risk assessment. The three-tiered homeland security risk analysis setting (Volume I) andsurvey of analytic methods, techniques, and tools (Volume II) contribute to theburgeoning dialogue and research applied to homeland security risk analyses that guideefforts to secure our homeland.



Documents

Risk Assessment Volume 1 Setting