D5.1 Security Testing Methodology - CRISALIS Projectcrisalis-project.eu/sites/crisalis-project.eu/files/crisalis... · D5.1 Security Testing Methodology Contract No. FP7-SEC-285477-CRISALIS

SEVENTH FRAMEWORK PROGRAMMETheme SEC-2011.2.5-1 (Cyber attacks against critical infrastructures)

D5.1 Security Testing Methodology

Contract No. FP7-SEC-285477-CRISALIS

Workpackage WP 5 - Vulnerability DiscoveryAuthor Marco Caselli, Frank KarglVersion 1.0Date of delivery M12Actual Date of Delivery M12Dissemination level PublicResponsible UT

The research leading to these results has received funding from the European Community’sSeventh Framework Programme (FP7/2007-2013) under grant agreement n°285477.

SEVENTH FRAMEWORK PROGRAMMETheme SEC-2011.2.5-1 (Cyber attacks against critical infrastructures)

The CRISALIS Consortium consists of:

Symantec Ltd. Project coordinator IrelandAlliander NetherlandsChalmers University SwedenENEL Ingegneria e Innovazione ItalyEURECOM FranceSecurity Matters BV NetherlandsSiemens AG GermanyUniversiteit Twente Netherlands

Contact information:Dr. Corrado Leita2229 Route des Cretes06560 Sophia AntipolisFrance

e-mail: [email protected]: +33 673 41 91 27

[email protected]

Contents

1. Introduction 61.1. General Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2. Standardization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.3. Testing Critical Infrastructures . . . . . . . . . . . . . . . . . . . . . . . . 10

2. Standard 122.1. OSSTMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.1. General information . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2. NIST SP800-115 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.2.1. General information . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.2.2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.3. ISSAF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.3.1. General Information . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.3.2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3. Requirements 373.1. Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.2. Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

A. General information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

B. Security requirements, design, and regulations . . . . . . . . . . . . . . . . 40

C. Testing and methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

D. Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

E. Follow up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.3. Further analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4. Methodology 504.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.2. Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.1. Main Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.2. Workflow phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3

4.2.3. Penetration Testing Steps . . . . . . . . . . . . . . . . . . . . . . . 574.3. Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5. Conclusion 63

4

Abstract

Existing security testing methodologies and tools cannot be applied directly to criticalenvironments, due to a number of safety and availability requirements. One of the goalof WP3 is to understand what assessment processes we can exploit in Industrial ControlSystem (ICS) environments and what are the constraints given by industrial componentsand infrastructures. This document provides an overview of assessment methodologiesused in standard IT. We analyze these methodologies to investigate their applicabilityin an ICS environment. We are also conducting a survey among stakeholders in orderto detail existing best practices adopted by companies to test their infrastructures anddiscover vulnerabilities in their systems. From this we are proposing a new methodologytailored to support security testing in Critical Infrastructure environments. As theevaluation of the feedback and the methodology is requiring more work than originallyforeseen and as our on-going investigations are also bringing up new aspects not foreseenin the original proposal (e.g., incident handling), this deliverable should be regarded onlyas a first version of D5.1 that we intend to extend in a version 2 either after year 2 orat the end of the project in order to address these additional aspects.

1. Introduction

Security testing involves various techniques which complement security engineering, i.e.,design and implementation of secure systems. In contrast, security testing implies testingof those systems with regards to their security properties. Security testing includesanalysis-centered activities like a code review but also practical tests to penetrate asecurity system and find vulnerabilities and implement attacks. The latter is oftentermed penetration testing.

Per definition, a penetration test is a simulation of an attack aimed at assessing thesecurity of a computer system or a network. Simulation refers to the fact that the testis conducted by the system owner or on his request. This test has multiple phases andallows to highlight infrastructures’ and protocols’ weaknesses. During a penetrationtest, the penetration tester tries to discover and sometimes exploit known or unknownvulnerabilities in order to obtain useful information or to access targeted hosts. At theend, all security problems discovered are reported to the system owner together with anassessment of their impact on the system. The analysis should also propose technical ororganizational mitigations to identified problems.

In its most simplistic form, a penetration test only involves running some automatedsecurity scanner, like, e.g., Nessus [1]. However, it is generally agreed that serious pene-tration testing typically involves also manual test planing, preparation, and conductionto be able to flexibly react to the system under test. In such a case, the penetrationtester should follow some structured methodology to ensure valid, reproducible, and welldocumented results.

The development and use of such manual penetration testing methodologies is moti-vated by several reasons. First of all, there are numerous vulnerabilities that may bedifficult or impossible to detect with automated tools. Network or application vulnera-bility scanning software are nowadays highly sophisticated. However, sometimes, highprofile systems and infrastructures need a comprehensive and systematic approach tobe compromised which cannot be achieved by automated tests. This is, e.g., related tothe detection of high-risk vulnerabilities resulting from a combination of low-risk ones.As multiple minor weaknesses are exploited in a particular sequences, a penetration testmay reveal all the possibilities identifying the possibly dangerous combinations.

More importantly than identifying vulnerabilities, determining the feasibility of a par-ticular set of attack vectors is another important target of a penetration test. As having

6

1.1. General Classification

theoretically unassailable systems seems impossible in practice, companies and organiza-tions are interested in running infrastructures that are “practically” secure. Therefore,the analysis becomes a way to rank vulnerabilities both by severity and realizability.

The potential business impact is a further valuable indicator of importance for a vul-nerability. Conducting practical attacks also aims at properly identifying necessary at-tacker effort which is a prerequisite for correct risk estimation. Assessing the magnitudeof economical and operational problems related to a successful cyber attacks is usuallythe last phase of analysis of a penetration test. Based on the knowledge of businessprocesses such analysis gives an accurate picture of the risk related to the infrastructureand to the work such infrastructure performs.

Finally, penetration tests are the experiments with which companies try out the abilityof network defenders to successfully detect and respond to cyber attacks. Their capacityof reaction is measured during the test and evaluated at the end of it. It will alsoprovide evidence to support increased investments in security personnel and technologyor it will suggest modifications in methods and strategies are the output of such kind ofassessments.

1.1. General Classification

Penetration tests can be performed in a broad variety of ways. The following criteriaprovide some orientation and classification.

The amount of information known by the fake attackers is one of the main discrim-inating factor of penetration tests. The simplest classification describes three differenttypes of tests: white-box, black-box, and gray-box.

White-box penetration testing outlines a situation in which fake attackers have fullknowledge of the system under attack. The goal of a white-box penetration test isto simulate the presence of an expert malicious insider. In some special cases suchinsider has also basic credentials for the target system. Among the information knownto the fake attacker we can have: network diagrams, domain names, IP addresses, butalso phone numbers, e-mail addresses. White-box testing of applications usually allowsintruders to have also programs’ source code. White-box test design techniques include[2]:

� Control flow testing

� Data flow testing

� Branch testing

FP7-SEC-285477-CRISALIS 7

1. Introduction

� Path testing

� Statement coverage

� Decision coverage

Black-box penetration testing is the opposite of the previous one. This method examinesinfrastructures’ and applications’ functionalities without knowing anything about them.In general, the tester is “aware of what the software is supposed to do but is not awareof how it does it” [3]. Thus, specific inputs (e.g network packets, shell commands, etc.)return specific outputs without the intruder knowing how the mechanisms within thesystem work. The goal of a black-box penetration test is to simulate a typical externalhacking or also cyber warfare scenario. Typical black-box tests design techniques include[4]:

� Decision table testing

� All-pairs testing

� State transition tables

� Equivalence partitioning

� Boundary value analysis

Finally, Gray-box penetration testing is a combination of white-box and black-box meth-ods. A gray-box tester partially knows the internal structure of the system under attack.This knowledge can include network topology, installed applications or also more in deepinformation like used algorithms and data structures. Since any combination of knowl-edge owned by the testers falls in the domain of gray-box penetration testing this methodis the most widely used, especially in the field of web applications. Gray-box testingtechniques include [5]:

� Matrix Testing

� Regression testing

� Pattern Testing

� Orthogonal array testing

8 SEVENTH FRAMEWORK PROGRAMME

1.2. Standardization

Other ways to characterize penetration tests include the type of system access (ex-ternal network, internal network, physical access), whether a test is conducted in purephysical ways or whether social engineering attacks are also performed, or whether test-ing includes performing (possibly disruptive) attacks or whether such attacks shouldonly be identified but not performed. The specifics of a test need to be carefully definedin the test plan.

1.2. Standardization

In this last decade the interest in penetration testing techniques has increased. Com-panies, organizations and governments provide numerous services on the Internet andneed to assess the security of their systems. Given the proliferation of different pene-tration testing methods, some organizations and standardization bodies decided to pro-pose comprehensive and usable methodologies. While many penetration testers followa “home-brewed” self-defined testing approach, following standardized approaches hassome general benefits.

A standardized and generally accepted penetration methodology provides a certainconfidence that no important steps or aspects of a test are forgotten by accident. As testsshould be repeated after significant changes to the system under test, following the samemethodology also provides the possibility to compare multiple tests to identify whethera system got more secure after mitigating measures have been taken. A standardizedapproach may also be required in cases where security tests are required from a legal orinsurance perspective. A standardized approach and common reporting forms also allowsto compare results with results from other installations. Finally, accepted standardssimplify negotiations between a security tester and clients about the scope of such tests.So there are a number of good reasons to look into such standardized and commonlyagreed approaches.

Over the years, many standardized methodologies have been proposed. From mostgeneral to system-specific ones, both governments and private organizations have pro-posed different ways to approach penetration tests. Among the most well-known pene-tration testing techniques are: OSSTMM (Open Source Security Testing MethodologyManual), NIST SP800-115, ISSAF (Information Systems Security Assessment Frame-work), OWASP (Open Web Application Security Project), and PTES (Penetration Test-ing Execution Standard). Later in this document, we are going to give a broad overviewof some of these approaches.

In addition to methodologies, different kind of systems exist to assess specific contextsof security testing without a structured approach. Companies can decide not to follow a


1. Introduction

methodology but just perform a light assessment using different methods. In this casesthey can exploit several guidelines and toolkits developed for numerous and differentpurposes. According to [6] toolkits “implement and bundle in a convenient package a setof testing techniques, usually aimed at discovering specific classes of security problems”.Guidelines instead “organize the process of security testing, by collecting sets of bestpractices, comprehensively listing items to be tested, and structuring any other kind ofuseful advice”. Finally, there are also collections of penetration testing tools such asBacktrack Linux or metasploit that are regularly used by penetration testers.

1.3. Testing Critical Infrastructures

Despite the recent interest for penetration testing methodologies, none of the proposedmethodologies specifically considers industrial control systems and critical infrastruc-tures despite the fact that penetration testing is also performed in such environmentson a routinely basis.

We argue that standard methodologies should not be applied there without specialconsideration. These systems and the networks that interconnect them show many dif-ferences compared with standard IT networks. Since critical infrastructures control andautomate real world processes or equipments which are potentially dangerous for hu-mans, any activity different from the main purpose including penetration testing shouldbe limited. So just running a full Nessus or Metasploit test in such an environment couldhave disastrous consequences.

There are three main concerns that make penetration testing on critical infrastructureschallenging. As introduced in the previous paragraph, any penetration attempt can havea consequence in the real world. What follows is an interesting example of such concerns.In a real incident, a gas utility conducting a penetration test on its corporate networkperformed this in a careless manner affecting some of the DCS systems. During thetest this DCS system was locked up by mistake and the utility was not able to sendgas through its pipelines for four hours leading to a large economic loss [7]. The strongrelation between the system under test and physical components that people dependon is the first important difference with respect to standard IT. This implies that anyCI penetration testing methodology needs to include safeguards to protect from suchfailures.

Another important difference concerns the devices used in Critical Infrastructure,these systems usually have very limited resources compared to devices found in normalIT environments. The longevity of network components is greater and the cycle ofreplacement of devices may also be in the order of decades. This implies that devices


1.3. Testing Critical Infrastructures

may be out of service, no patches may be available, and generally proposing mitigationstrategies may take different approaches. This also means that critical infrastructurescan be more easily overloaded with respect to other IT systems. A penetration testingmethodology has to take care not to override the normal operation of every componentin the ICS unless identifying DoS attacks is specifically in the scope of the test.

Finally, due to the size of most of the critical infrastructures, a penetration testing hasalways to check if localized attacks have effects in different parts of the network whichmay be hard to assess due to indirect effects. At the end of a test the identification ofa vulnerability is followed by the calculation of the damage that it can cause. Linkingthese two elements in a system with thousands of components requires the assistanceof experienced personnel both in the IT field and in the field related to the industrialprocess under control. As a consequence, penetration tests cannot be conducted bysecurity experts in isolation.

We think that a penetration testing methodology for critical infrastructure must ex-plicitly describe how to deal with the previous issues. Moreover, it has to identify specificmethods, tests, and tools already used in standard penetration testing techniques andexplain how to tune these in order to be suitable for industrial control systems. Finally,such an ICS penetration testing methodology must provide – where necessary – newinstruments and customized methods to test IT and networking components not foundin standard ICT (e.g., Process Control Units, Remote Terminal Units, field networks,etc.).


2. Standard

In this chapter, we are going to review multiple penetration testing methodologies pro-posed by standardization bodies and other organizations. The aim is to provide a broadoverview over the state of the art before defining specific requirements for CI penetra-tion testing and then proposing an own CRISALIS penetration testing methodology forCritical infrastructures that is distilled from those existing approaches.

2.1. OSSTMM

2.1.1. General information

The Open Source Security Testing Methodology Manual (OSSTMM) [8] is an open stan-dard methodology for security tests. Developed by Pete Herzog at the end of 2000 asan ethical hacking framework, it has rapidly grown to become a comprehensive method-ology to assure security at operational level. Version 3, released in 2008, encompassestests for every security aspect: from personnel qualification to physical security, fromcontrol of communication to electronic systems safety. As every standard methodology,it is designed to be consistent and repeatable. Moreover, it is openly available and thusallows a free dissemination and free use.

Figure 2.1 shows a comparison among OSSTMM and common security tests in termsof accuracy and thoroughness. By its nature, the methodology can be adapted to anykind of audit operation (e.g. penetration tests, ethical hacking, security and vulnera-bility assessments, red/blue teaming, etc.). The primary purpose always remains thedescription of a security examination path and the definition of a way to consistentlycorrelate test results.

The usage of the OSSTMM allows the auditor to perform a validated set of tests andto qualify its examination as a “certified OSSTMM audit”.

Differently from other methodologies, mitigations to security flaws are not requiredas part of an OSSTMM audit. Recommendations may well be part of the final results,but common operational situations often imply that auditors have a limited view andknowledge of the client environment. In these cases it is difficult to propose properand feasible solutions. For this reason solutions are usually avoided in final reports butshould we worked out together with the system owner after the test. This is an aspect

12

2.1. OSSTMM

Figure 2.1.: OSSTMM comparison with common security tests (from [8])

that is well in line with our findings from the previous chapter as it will also apply toCI.

An important aspect of the OSSTMM is the compliance with general security policies.The methodology is developed taking into account major legislation and regulations.Furthermore, OSSTMM defines three different types of compliance and a set of rules todeal with all of them. The three types of compliance defined in the methodology are:

� Legislation: enforces the security test to be compliant with region/country legis-lation. The lack of this requirement could lead to criminal charges.

� Regulation: enforces the security test to be performed accordingly to standardregulation within industry sector. A failure in this task could lead to a dismissalof the company from the group.

� Policy: enforces the security test to be done according to business and organizationpolicies. Violation could cause a dismissal of the responsible from the company.

OSSTMM provides a list of national and international regulations already proved tobe compliant to the proposed methodology.


2. Standard

2.1.2. Overview

Before describing the workflows, the OSSTMM introduces several concepts to outlinewhat kind of security tests can be performed and what the scope of such tests can be.

Figure 2.2 presents the six different security test types defined by OSSTMM. Possiblesecurity tests are however not limited to these cases and can be tuned to more specificrequirements.

Figure 2.2.: OSSTMM security test types map (from [8])

� Blind: this type of security test describes a situation in which the attacker does notknow anything about the target. On the other side, the defender knows everythingabout the audit operations and it is prepared to take countermeasures. A blindsecurity test is primarily used to evaluate attacker skills.

� Double Blind: as before the attacker does not know anything about its target but,this time, the target is not notified in advance about the security test. A doubleblind audit is used to test both the skills of the attacker and the preparedness ofthe defender.

� Gray Box: in a gray box security test the attacker has limited knowledge of thetarget and can better engage the defender. The defender still has the advantageof full knowledge about the audit in progress. As in the blind security test, thiscase is mostly used to evaluate attacker’s skills.


2.1. OSSTMM

� Double Gray Box: in this type of security test both the attacker and the defenderhave limited knowledge about their counterparts. Such situation is used to evaluateboth the contenders. With respect to double blind this test increases the level ofaccuracy and pervasiveness.

� Tandem: in the Tandem security test the attacker has full knowledge of thetarget and the defender is fully prepared to bear the test. Such situation is usedto improve the thoroughness of the audit as the attacker has a full view of all testsand their responses.

� Reversal: as in the previous situation, the attackers knows everything about thetarget while the defender knows nothing about its contender. The purpose of thistest is to evaluate every operational aspect of the defender.

Once the test type of interest is identified, it is necessary to identify the scope ofthe audit. OSSTMM divides such scope into three different channels: COMSEC (com-munication security), PHYSSEC (physical security), SPECSEC (spectrum security). Athorough security audit would require testing all three channels but usually the tests aretuned on more specific needs. For this reason, the methodology makes a further parti-tioning, addressing these three channels into five logical sections. Channel PHYSSEC isdivided into “Human” and “Physical”, channel SPECSEC contains “Wireless Commu-nication”, and channel SPECSEC is divided into “Data Networks” and “Telecommuni-cations”.

What follows is a description of the five logical sections:

� Human: concentrates on the human element and its communication processes.

� Physical: focuses on physical security testing and comprises tangible elementsof security where interaction requires physical effort or an energy transmitter tomanipulate.

� Wireless Communications: concentrates on security tests on wireless electroniccommunications and signals.

� Data Networks: comprises tests on electronic systems and data networks.

� Telecommunications: focuses on digital/analog telecommunications where in-teraction takes place over established telephone network lines.

The full methodology is divided into seventeen modules. Every module defines aspecific target and several tasks that are needed to achieve it. Each module has an input


2. Standard

and an output. The input represents the information needed to perform each task whilethe output is the result of the accomplished tasks. Figure 2.3 shows the methodologyflow.

OSSTMM uses the same workflow for all the channels. The seventeen modules andthe methodology flow apply in the same way to the five logical sections. However, whilethe methodology remains the same, each channel will differ in the proposed tasks.

The methodology is divided into four phases: Regulatory Phase (modules 1 to3), Definitions Phase (modules 4 to 7), Information Phase (modules 8 to 13) andInteractive Controls Test Phase (modules 14 to 17). Each phase faces a differentlevel of depth in the audit. The regulatory phase focuses on understanding the auditrequirements, scope and constraints. The definitions phase improves the description ofthe scope and of the inner interactions and processes. The information phase uncoversmisplaced and mismanaged information flows in the target. Finally, the interactivecontrols test phase focuses on practical penetration testing. This is often the final phaseof a security test. This ensures that practical attacks do not affect less invasive tests.

In what follows we briefly describe the focus of every module.

� Posture Review: it is used to define the scope and identify what tests mustbe performed. This module focuses on rules, norms, regulations, legislations andpolicies applicable to the target.

� Logistics: it defines which limitations the audit process has. It relates to interac-tion constraints such as physical distance, communication speed, etc.

� Active Detection Verification: it takes into account any practical restrictionimposed against performing interactive tests on the targets.

� Visibility Audit: it determines available targets within the scope. This moduleprovides the first in depth description of the targets and how they interact withthe scope.

� Access Verification: it measures the breadth and depth of interactive accesspoints. It verifies if such access point to the target exists and if it requires authen-tication.

� Trust Verification: it identifies trust relationships from and between the targets.This module verifies if targets accept interaction freely and without credentials.

� Controls Verification: it measures the effectiveness of process-based attributeslike non-repudiation, confidentiality, privacy and integrity.


2.1. OSSTMM

Figure 2.3.: OSSTMM methodology flow (from [8])


2. Standard

� Process Verification: it verifies the existence, effectiveness and maintenance ofsecurity levels defined for the targets.

� Configuration/Training Verification: it explores the default conditions underwhich targets operate regularly to understand their intents, business justificationand reasoning.

� Property Validation: it checks the use of illegal or unlicensed intellectual prop-erty or application within the targets.

� Segregation Review: it checks the amount of personally identifiable informationavailable depending on the applied rules and legislations.

� Exposure Verification: it searches for freely available information which uncov-ers indirected or unwanted visibility of the targets.

� Competitive Intelligence Scouting: it searches for freely available informationwhich could harm or adversely affect the target.

� Quarantine Verification: it determines the effectiveness of authentication interms of black and white listing quarantines.

� Privileges Audit: it measures the impact of misusing credentials and privilegesand the consequences of unauthorized escalations of privileges.

� Survivability Validation/ Service Continuity: it checks the resistance of thetarget to excessive or adverse changes.

� Alert and Log Review/End Survey: it is a review of the performed auditactivities.

The outcome is a comprehensive assessment of the tested channels. Results are pre-sented through Risk Assessment Values (RAVs) that are a set of security metrics pro-viding an overview on the state of the channels (e.g. measurement of the attack surface,the amount of uncontrolled interactions with a target, etc.). To finally measure thethoroughness of the test performed, the OSSTMM suggests to conclude the work witha Security Test Audit Report (STAR). This step requires to record in the appropriateform the following information:

� Date and time of test

� Duration of test


2.2. NIST SP800-115

� Auditors and analyst involved

� Test type

� Scope of test

� Test Index

� Channel tested

� Test Vectors

� Verified test and metrics calculations of the operational protection levels, loss con-trols and security limitations

� Knowledge of which tests have been completed, not completed, or only partiallycompleted, and to what extent

� Any issue regarding the test and the validity of the results

� Test error margins

� Any processes which influence the security limitations

� Any unknowns anomalies

Overall, OSSTMM provides a very comprehensive framework that addresses a largevariety of practical aspects. It is specifically useful to ensure completeness of our laterapproach. However, it is also relatively generic and is not going into details of specificsystems and is not addressing the CRISALIS scenarios at all.

2.2. NIST SP800-115

2.2.1. General information

The National Institute of Standard and Technology (NIST) has designed its own securityassessment methodology in the Special Publication 800-115 [9]. The purpose of thispublication was to provide guidelines for organizations on planning, conducting andevaluating information security testing. Even if the overall goal is to propose an overviewof main key elements of technical security assessments, NIST SP800-115 provides alsopractical recommendations and technical information relating to penetration tests.


2. Standard

NIST defines the security assessment as “the process of determining how effectivelyan entity being assessed meets specific security objectives”. From an high-level point ofview, there are three possible ways to accomplish such target:

� through Testing, by exercising the assessment objects comparing actual and ex-pected behaviors

� through Examination, by analyzing the assessments objects and understand theirfunctioning

� through Interviewing, by conducting extensive discussions with organizationsand experts to identify possible problems

A methodology that implements one of the previous points has to contain at leastthree different phases:

� Planning: this phase is necessary to collect all the information needed for the as-sessment. This concerns elements like assets, threats, policies in use, etc. Moreover,the assessment should have a final goal and a set of objectives to work towards.

� Execution: this phase concerns the operational part of the assessment. The Exe-cution implements operations of environment discovery and analysis, vulnerabilityidentification and result validation. Practical activities for this phase obviouslydiffer from case to case. However, the main goal is to bring to light problems attechnical and organizational level.

� Post-Execution: this phase describes all the analysis aim at identifying vulner-ability causes and mitigation strategies. The development of a final report is themain output of the Post-Execution.

As already discussed, NIST SP800-115 offers an overview of security assessment keypoints. However it suggests several practical techniques to implement penetration test-ing. According to the authors, the most commonly used pentesting techniques can begrouped in the following categories:

� Review Techniques define how to examine and analyze assets and their securitypolicies.

� Target Identification and Analysis Techniques define how to identify assess-ments targets and how to describe their properties.


2.2. NIST SP800-115

� Target Vulnerability Validation Techniques: define how to verify the pres-ence of vulnerabilities within the targets and how to test their security levels.

The NIST methodology focuses on how these techniques can be used together, butdoes not provide a list of technical activities to be used in any specific circumstance. Inaddition to practical penetration testing techniques, the methodology proposes severalnon-technical methods that can be used to enforce the assessment (e.g. physical securitytesting).

Finally, NIST SP800-115 concentrates on how singular tests and final reports can becompared with each others. The possibility to compare two security assessments relieson the correct representation of the Testing Viewpoint. Every test can be performedfrom different viewpoints. This implies different information owned by the auditors ordifferent physical location of the pentesters. Results of comparisons among differentassessments always need to consider the kind of testing. NIST SP800-115 defines fourtesting types:

� External Security Testing: this type of testing is conducted from outside theorganization’s security perimeter. The goal of this security test is to reproduce theview of an external attacker and to draw the attention to vulnerabilities alreadyvisible from outside the company premises (e.g. from the Internet). Externaltesting always begins with discovery techniques aiming at examining organizationpublic information.

� Internal Security Testing: differently from the previous one, this kind of testingis performed within an organization’s perimeter (e.g. internal network). Thepurpose of the Internal Security Testing is to impersonate a trusted insider or alsoan attacker that has penetrated external defenses. This test allows pentesters tohave some level of access to the network and, thus, to internal information. Theseelements make it possible to assess internal security mechanisms.

� Overt Security Testing: also known as White Hat testing, defines an Internalor External Testing in which pentesters have a full knowledge of organization’ssystems and processes. Company staff is also aware of the test and it usually triesto limit the testing impact. This kind of test is often used as company training.

� Covert Security Testing: also known as Black Hat testing, uses an adversarialapproach by not giving any knowledge about the company to the pentesters. Inthis case, company IT staff is usually also not aware of the test and its responseis used for testing the technical and organizational security controls within thecompany.


2. Standard

2.2.2. Overview

In this section we describe in detail the three groups of techniques discussed before andthe three phases into which the methodology is logically structured.

Review Techniques describe the process of passively gathering information relatedto the organization and the consequent analysis operations. This information regardsboth policies and equipments (e.g. systems, networks, etc.). Since these techniques arepassive, they do not interfere with company operations and systems. For this reason,this is usually the first phase in every security assessment. Possible review techniquesare:

� Documentation Review examines the technical aspects of used polices and pro-cedures. Security flaws or weaknesses in such elements can lead to missing orimproperly implemented security controls. The results of the analysis can also beused to better tune the following techniques.

� Log Review determines if important information and operations are properlylogged and recorded. This is a necessary element to prove that all the systems areworking according to the applied policies and procedures. Log analysis can showalso misconfigured systems and malicious operations (e.g. attempted or successfulattacks).

� Ruleset Review checks the collection of rules and signature-based security mech-anisms used to detect malicious operations or implement organization’s processes.Routers, firewalls and intrusion detection systems are the main targets of this anal-ysis. Router access control lists are checked to identify whether rules are no longerneeded or enforce the refusal of unknown traffic. The review refines in depth fire-wall rules by checking unnecessary open ports and allowed services. Moreover, itsearches for possible backdoors and malware. Finally, IDS rulesets are verified tobe up to date and tuned to organization’s needs.

� System Configuration Review searches for weaknesses in systems and applica-tions configurations (e.g. programs not enforcing security or non-compliant withsecurity policies). This review usually concerns: unnecessary services running onservers, improper user account and password settings, etc.

� Network Sniffing implements the passive monitoring of a data network. It canrely on communication flows, protocols’ decoding, and header/payload analysis.It is used to map organization communication infrastructures and gathering infor-mation about used systems, applications and services. It can also identify unau-


2.2. NIST SP800-115

thorized and inappropriate activities or capture unencrypted personal information(e.g. usernames and passwords).

� File Integrity Checking provides a way to check file modifications. It is usuallyimplemented through checksum.

Target Identification and Analysis Techniques focus on identifying active devices throughopen ports and running services and analyze them looking for vulnerabilities. NISTSP800-115 discusses several different techniques belonging to this group:

� Network Discovery concentrates on discovering active and responding host on anetwork. Active techniques send various types of packets to the network waiting forresponses. This system is usually an improvement of the passive analysis discussedin the Review Techniques as it is able to find hosts not currently involved in anycommunication. During this analysis, pentesters have to face firewalls and intrusiondetection systems that usually identify many instances of scans. Active discoverygenerally produce network noise and communication delays.

� Network Port and Service Identification is usually performed in parallel withthe network discovery. This check involves the use of port scanners able to un-derstand services and application running on remote hosts. Information gatheredduring the process allows pentesters to exploit OS fingerprinting tools. Such appli-cations guess operating systems working on the network and provide a descriptionabout their versions and updates.

� Vulnerability Scanning, like the previous two operations, collects informationabout hosts in the network. Moreover, it also attempts to identify specific vulner-abilities on discovered hosts. Vulnerability scanners usually provide: a check onthe compliance of security policies, a list of targets for the following penetrationtesting, and preliminary information on how to mitigate discovered vulnerabilities.

� Wireless Scanning: as the number of wireless devices is increasing, these tech-niques are becoming more and more important. Wireless scanning techniques canbe further divided in: passive scanning, active scanning, bluetooth scanning, anddevice location tracking. An example of passive scanning involves the analysis ofMAC addresses (e.g. vendor identification or white/black listing of devices). Anactive scanning can be used instead to verify authentication mechanisms and dataencryption. Bluetooth security issues have been extensively discussed in litera-ture [10], [11]. Bluetooth scanning is more difficult than wireless one due to theits short range. However, a confirmation of compliance with organization security


2. Standard

requirements is needed to have a comprehensive security assessment. Finally, wire-less networks can be used to locate organization’s devices. This check enforces thereconstruction of the company’s network topology.

Target Vulnerability Validation Techniques uses the information found in the two pre-vious phases and further explore the existence of potential vulnerabilities. The target isusually not limited to identifying the vulnerability but also to show the security exposureby trying to exploit it. This procedure is often risky for the organization infrastructureand processes. For this reason is usually the last active work performed on the network.In this way, it will not compromises other tests. NIST SP800-15 defines and implementsits penetration testing method in this phase. The targets of the penetration testing canbe numerous. Pentesters are interested in understanding how well the system toleratesreal attacks, identifying necessary countermeasures, and evaluating defenders’ abilities.Moreover, this attack simulation can show what level of skills attackers need to compro-mise organization’s systems. The proposed penetration testing is divided in four stepsas show in Figure 2.4.

Figure 2.4.: NIST SP 800-115 penetration testing steps

The Planning step prepares the necessary information and background for the attack.The Discovery step integrates information already owned by pentesters with a newsession of network scanning. With respect to the other phases, the attackers try toacquire information about: host names and IP addresses, employee names and othercontacts, and system information (e.g. names and shared resources). In some cases alsotechniques as “dumpster diving” can be used to enforce the process. The Discoverystep improves also the vulnerability analysis and finalizes the list of targets. The Attackstep is the core of every penetration testing. During the attack a continuous feedback


2.2. NIST SP800-115

is provided back to the discovery step as succeeding to access a system gives pentestersadditional knowledge of the infrastructure under attack. Figure 2.5 shows an exampleof attack.

Figure 2.5.: NIST SP 800-115 attack example (from [9])

NIST SP800-115 organizes the most common vulnerabilities exploited during the At-tack step in the following categories:

� Misconfiguration: misconfigured security settings (e.g. insecure default set-tings).

� Kernel Flaws: security flaws in the core code implementing an operating systemthat can endanger the whole system.

� Buffer Overflows: inadequate checks on input sizes. This misbehavior can causearbitrary code been executed with the privileges of the running program.

� Insufficient Input Validation: missing checks on the content of input strings,files, etc. Network applications without filters can run malicious commands (e.g.SQL injection).


2. Standard

� Symbolic Links: as the operating system allows to change permissions related tofiles, a malicious user can trick the system by creating symbolic links and modifyingpermission passing through them.

� File Descriptor Attacks: malformed file descriptors can grant access to therelated files.

� Race Conditions: an attacker can take advantage of high privileges processes bymodifying their execution order and manipulate results.

� Incorrect File and Directory Permission: incorrect permissions can causevarious lacks of information (e.g. reading and writing password files).

The last step, called the Reporting, occurs simultaneously with the other three stepsand records all the results. At the end of the test, pentesters report the identifiedvulnerabilities, provide a risk rating, and propose possible mitigation techniques fordiscovered weaknesses.

Security Assessment Planning provides information on creating an assessment policy,scheduling tests, identify the most suitable approach, and addressing possible problems.It also enforces assessments to be compliant to current regulations and legislations.According to NIST SP800-115 this planning activity should identify:

� Organizational requirements assessments have to take into account

� Roles and responsibilities

� Adherence to already established methodologies

� Assessment frequency

� Documentation requirements

In this phase organizations have to choose assessment techniques and customize themto fit the requirements. Organizations should also carefully consider risks related to usesuch techniques on their infrastructures. The selection of auditors and the identificationof needed skills is another fundamental step of the planning. Assessors experience candetermine the success or the failure of the evaluation. Organizations can choose the typeof assessment (e.g. External/Internal and Overt/Covert) and finally propose technicaltools and resources available to the assessors.

At the end, the assessment plan should answer to the following basic questions:


2.2. NIST SP800-115

� What is the scope of the assessment?

� Who is authorized to conduct the assessment?

� What are the assessment’s logistics?

� How should sensitive data be handled?

� What should occur in the event of an incident?

Security Assessment Execution phase highlights key points for assessors to considerthroughout the execution phase.

Assessors have to be coordinated with various entities and stakeholders within theorganization. Appropriate organization’s personnel should help and supervise the as-sessment for the whole duration of the activity. During operations assessors can facedifferent kind of problems (e.g. technical, operational, political, etc.). These can in-clude:

� Resistance: to the assessments, from entities within the organization (e.g. net-work administrators or end users).

� Lack of Realism: that indicates the preemptive modification to system settingsin order to make the infrastructure more secure to pass the tests.

� Immediate Mitigation: that describes the adjustment and the resolution on-the-fly of weakness found during the assessment. This can disturb following testsand compromise the final report.

� Time: time constraints are usually a problem as security assessments are oftenan exception more than a normal operation integrated in the development anddeployment cycle.

� Resources: organization can make a stand on providing and maintaining adequateresources for security assessments.

� Evolving Technology: tools used for security assessments as well as used tech-niques can be out-of-date.

� Operational Impact: even if a security assessment should prevent any impacton infrastructure functioning, there is always a chance of accidental problems. Forthis reason, an incident response plans should be already in place during testing.


2. Standard

Assessors can make some preliminary analysis on the assessment during the assessmentitself. This preliminary analysis of the causes of the noticed problems becomes the inputfor the last phase of the security assessment. Most common problem causes can beorganized in the following sets:

� Insufficient patch management

� Insufficient threat management

� Lack of security baselines

� Poor integration of security into the system development life cycle

� Security architecture weaknesses

� Inadequate incident response procedures

� Inadequate training for end users and network/systems administrators

� Lack of security policies or policy enforcement

Finally, the Post-Testing Activities concentrates on finalizing the analysis and propos-ing mitigation actions and recommendations. Security testing reports should be used bythe organization as follows:

� As a reference point for corrective action

� In defining mitigation activities to address identified vulnerabilities

� As a benchmark for tracking an organization’s progress in meeting security re-quirements

� To assess the implementation status of system security requirements

� To conduct cost/benefit analysis for improvements to system security

� To enhance other life cycle activities (e.g. risk assessment)

� To meet reporting requirements

Overall, NIST SP800-115 also provides a well-structured and complete approach thatgoes into more technical detail than OSSTMM. From this perspective, it is interestingto discuss whether such details are necessary and helpful and make a methodology moreapplicable or whether they just inevitably restrict its use to certain use cases.


2.3. ISSAF

2.3. ISSAF

2.3.1. General Information

The Information Systems Security Assessment Framework (ISSAF) [12] is a peer re-viewed structured framework designed by the Open Information Systems Security Group(OISSG). The methodology defined by ISSAF covers all the aspects related to securityassessments: from an high-level perspective (e.g. business impact and organizationalmodels) to practical techniques (e.g. security testing of passwords, systems, network,etc.).

The framework is divided in four main phases structured in several working packages(named “activities”). The four phases are respectively: Planning, Assessment, Treat-ment, and Accreditation. Figure 2.6 shows ISSAF workflow.

Figure 2.6.: ISSAF framework overview (from [12])

The Planning phase concerns all the operation needed to define a project plan for the


2. Standard

whole assessment process. Before proceeding with the operational part, organizationshave to motivate the assessment to substantiate the underlying concerns. Moreover,organizations have to prepare a business plan to cover costs and identifying availablepersonnel and resources. The main steps in this phase are:

� Information Gathering

� Project Chartering

� Resource Identification

� Budgeting

� Cash Flow

� Work Breakdown Structure

� Project kick-off

The Assessment phase defines the approach needed to evaluate the Information Secu-rity Risks within an enterprise. This phase focuses on the risk assessment process andaddresses every component involved. It is divided into two categories: Inherent RiskIdentification and Controls Assessments. The first consists of the following activities:

� Identification of Assessment entities: during this activity the organizationspots the targets of the assessment (e.g. processes, assets, facilities, etc.).

� Identification of Threats and Vulnerabilities: this activity describes how tosearch, list and describe organization’s vulnerability.

� Impact Assessment: this activity measures the impact of an exploited vulnera-bility to the business of the organization

� Likelihood Assessment: during this activity the organization evaluates the like-lihood that a vulnerability is exploited.

Control Assessment explores instead the following fields:

� Evaluation of Legal and Regulatory Compliance: this activity evaluates theposition of the organization with respect to current regulations and legislations.


2.3. ISSAF

� Evaluation of Enterprise Information Security Policy: this activity focuseson understanding current organization’s approach on implementing and maintain-ing its security policies.

� Evaluation of Enterprise Information Security Organization and Man-agement: this activity directly follows the previous one, as a review on how theInformation Security is organized inside the company.

� Assessment of Enterprise Information Systems Security and Controls:this activity is in charge of reviewing different security aspects of the organization.For example: physical security, environmental security, technical controls (e.g.network security, host security, etc.), social engineering, etc.

� Evaluation of Enterprise Security Operations Management: during thisactivity the organization gains an understanding of the risk of specific operationsprocesses. The operational areas involved in this operations are: Capacity Man-agement, Vulnerability Management, User Management, etc.

� Evaluation of Enterprise Business Continuity Management: this activityevaluates organization capability of ensuring continuous availability of the Infor-mation Technology infrastructure.

The output of the Assessment phase is reported within the Inherent Risk and theResidual Risk documents.

The Treatment phase implements a platform for taking decision about the afore-mentioned risks. Its work-flow goes through the selection of safeguards, the develop-ment of implementation plan for countermeasures, and the development of a decisionmaking process. In this phase the organization decides to accept, mitigate, or avoidvulnerabilities and associated risks identified in the previous phase.

Finally, the process of Accreditation defines the steps needed to an organization toobtain the ISSAF certification.

2.3.2. Overview

In this section we will review the Penetration Testing methodology proposed by IS-SAF leaving aside all practical details regarding specific network components assessingmethods.

ISSAF penetration testing methodology evaluates organization’s network and systems.It consist of three phases and nine assessment steps. The three phases are: Planningand Preparation; Assessment; Reporting, Clean-up and Destroy Artifacts.


2. Standard

The Planning and Preparation phase comprises all the assessment’s preliminary opera-tions. These regards: requirements definition, actions to be performed, and expectations.ISSAF requires the identification of a contact person from the target company and thosewho will perform the tests. During the kick-off meeting both parties have to agree on thespecific engagement team, the exact dates, times of the test, escalation path and otherarrangements. The meeting has to be concluded with the signing of a formal AssessmentAgreement that will provide mutual legal protection.

The Assessment phase is the core of the penetration testing methodology. ISSAFapproach is a path that leads the attacker to nine operation steps as shown in Figure2.7.

� Information Gathering: ISSAF defines the information gathering in the broadersense possible. It comprises both technical and non-technical methods. The for-mer looks for DNS records and exploit tools like WHOIS. The latter implementssome social engineering methods (e.g. using search engines and mailing lists to findinformation about the target). The more a pentester knows about his victim themore he will improve his chances to successfully attack it. During this phase, theattacker does not always need to establish a contact with the target. Importantinformation can be collected also from public sources (e.g. the Internet). As secu-rity assessments are generally limited in time and resources, information gatheringis important to identify points that will be most likely vulnerable and focus onthem.

� Network Mapping: this phase directly follows the previous one and refines theinformation gathering with a more technical approach. The target is to identifywhat devices are working in the network and outline its topology. Network mappinginvolves the use of fingerprinting and network analysis tools. More in details,ISSAF specifies different classes of actions to be performed:

– Find live hosts

– Port and service scanning

– Perimeter network mapping (e.g. router, firewalls)

– Identifying critical services

– Operating System fingerprinting

– Identifying routes using Management Information Base (MIB)

– Service fingerprinting


2.3. ISSAF

Figure 2.7.: ISSAF pentesting overview (from [12])


2. Standard

At the end of this phase, pentesters will have the possibility to confirm or dismisssome preliminary made hypotheses regarding target systems.

� Vulnerability Identification: once specific targets are selected, the attackerswill perform several activities to detect actual exploitable vulnerabilities in thesystem. ISSAF lists some of the activities used to detect weak points:

– Identify vulnerable services using service banners

– Perform vulnerability scans

– Perform false positive and false negative verification

– Enumerate discovered vulnerabilities

– Estimate probable impact

– Identify attack paths and scenarios for exploitation

� Penetration: this phase is the first actual operation against the target. The at-tacker will try to circumvent security measures and gain access to the system underattack. ISSAF methodology emphasizes the following steps within the penetration:

– Find proof of concept code/tools to use for the attack

– Develop new tools/scripts if needed

– Test proof of concept code/tools before use them to avoid problems duringthe actual pentesting

– Use proof of concept code against target

– Verify or disprove the existence of vulnerabilities

– Document findings

� Gaining Access & Privileges Escalation: in this phase pentesters will confirmthe possibility to access the target. To do that the attacker has to compromise allthe intermediate systems in the path leading to the physical system under attack(e.g. firewalls, routers, etc.). Moreover, the attacker will usually access the systemwith the least privileges possible and he will try to obtain administrator’s rightsexploiting local vulnerabilities with root-kits.

� Enumerating Further: this phase lists malicious activities the attacker can per-form within the compromised system. Such activities are:

– Obtain encrypted passwords for off-line cracking

– Obtain passwords by using sniffing or other techniques


2.3. ISSAF

– Sniff traffic and analyze it

– Gather cookies and use them to exploit sessions and for password attacks

– E-mail address gathering

– Identifying new routes and networks

– Mapping internal networks

� Compromise Remote Users/Sites: this phase focuses on the presence of securecommunications (e.g. Virtual Private Networks) between remote users/sites andenterprises network. In this scenario, the attackers can try to extend his controlto remote users and detached networks. In case of success, the attacker will repeatall the previous steps in the new environment.

� Maintaining Access: in this phase pentesters make sure to control the systemalso in the future. As a vulnerability can always be patched by system operatorsan attacker can install malicious code in the system to create covert channels andbackdoors. Such verification shows the degree of exposure that a system can haveonce compromised. The Maintaining Access phase is often not performed as partof a penetration test due to the risks involved (e.g. a real attacker capable to takepossession of the covert channel and gain access to the system).

� Covering Tracks: this phase ensures attackers to be invisible to later analysis.Usually, during a penetration testing, attackers should be as open as possibleabout their operations (unless the company expressly requests otherwise). Thereare several actions that pentesters can perform to hide their traces:

– Hide modified files and used tools

– Clear logs by checking history and edit log files

– Defeat integrity checking

– Defeat Anti-viruses

– Implement customized Root-kits

� (optional) Audit: system audits should be performed after completing a penetra-tion test. This phase can usually bring to light further potential vulnerabilitieswithin the system.

The Reporting, Clean up and Destroy artifacts phase concludes the assessment. Ac-cording to ISSAF a minimal reporting should consist of:


2. Standard

� Verbal Reporting: that describes the feedback the company has to have imme-diately after the pentesting. The attackers will give to the company an overviewon their finding and they will focus on major critical issues.

� Final Reporting: that is an in depth presentation with detailed results of thetests. Each discovered vulnerability has to be presented with related countermea-sures and a proposal on how to implement them. The report should follow awell documented structure. ISSAF proposes a list of information needed for thedocument:

– Management Summary

– Scope of the project

– Tools that have been used

– Dates & times of the actual tests on the systems

– Outputs of tests performed

– A list of all identified vulnerabilities with included recommendations on howto solve problems found

– A list of action points

Finally, all information that is created and/or stored during the test on target systemsshould be removed. If this is not possible, such information (e.g. files and directories)should be mentioned to system operators to proceed to their removal.

This concludes the discussion of existing penetration testing methodologies. In a laterversion, we may extend this to other approaches like OWASP, PTES, or CHECK.


3. Requirements

3.1. Approach

Within the CRISALIS project we merge the expertise of different stakeholders. Academia,SCADA vendors, and SCADA operators are three of the main contributors to our assess-ment methodology for critical infrastructure. However, to further enlarge the informationbasis that our approach is built upon, we decided to create a questionnaire about testingsecurity infrastructure. The purposes of interviewing CI and security experts are several.There is the need to understand what are the requirements and the expected feedbacks.Moreover, we also need to validate the benefits of establishing a methodology tuned toCIs and also evaluate its comprehensiveness and accuracy. Using a questionnaire seemedthe most efficient way to reach a broader community and inquire about the state of theart of CI testing in general and to identify possible steps forward toward the definitionof the more comprehensive concept of “CI vulnerability assessment”. The questionnaireis then extended by more detailed interviews with some stakeholders.

The questionnaire is divided into four sections: General information; Security require-ments, design and regulations; Testing and methodologies; and Research.

The General Information section focuses on characteristics of the interviewed per-sons. Information on area of expertise and experience will be used to evaluate the laterresponses and comments.

The security requirements, design and regulations section collects information aboutdeployed security mechanisms and authentication schemata. Moreover, the section in-vestigates on what limitations there are in deploying security mechanisms in CriticalInfrastructures.

The testing and methodologies section is the core part of the questionnaire. It aimsto identify currently used security practices focusing on: kind of tests performed, testfrequency, and tools used. Furthermore, the section asks about the expectations theexperts have from a CI security methodology.

Finally, the Research section concludes the questionnaire asking about CI securityresearch status.

We believe that integration of CRISALIS expertise and questionnaire outputs will pro-vide the necessary background from which to build a comprehensive security assessment

37

3. Requirements

methodology for critical infrastructure. Such a methodology is supposed to take into ac-count already existing best-practice provided by industry and different, but relevant, ITfields. The output has to be compliant with CI experts and operators needs and shouldallow them to have a common evaluation framework to compare their infrastructures.

Section 3.2 presents the questionnaire we used to collect information. The question-naire is also available online at Critical Infrastructure Security & Penetration Testing[13].

3.2. Questionnaire

Critical Infrastructure Security & PenetrationTesting

Industry/Researcher Questionnaire

University of Twente

December 2012

This questionnaire is anonymous unless you want to participate in the follow up men-tioned in section E. The purpose of this work is exploring the need for a standard pene-tration testing methodology for Critical Infrastructure. With the following questions wewill collect ideas and requirements useful to realize such methodology.

N.B. If you have never worked with Critical Infrastructure please fulfill sections B andC with your opinion on how these systems are supposed to work and be tested. Questionsmarked with ‘*’ are intended only for Critical Infrastructures operators/deployers.

A. General information

1. In which area are you working?

# SCADA Vendor

# SCADA End User

# SCADA Consultant


https://docs.google.com/spreadsheet/viewform?formkey=dG15cUV5ck9HWlg1eTVuVk51ZnNYb1E6MQ#gid=0

A. General information

# Security Consultant

# Academia

# Other:

Comments:

2. What is your domain of interest? (Multiple selections are possible)

2 Water

2 Energy

2 Oil & Gas

2 Transportation

2 Other:

Comments:

3. Briefly describe your field of work/research and your job.

4. How much experience do you have in working with Critical Infrastruc-tures (CIs)?

(little) 2—2—2—2 (a lot)

Comments:

5. How much experience do you have in security?


Comments:


3. Requirements

6. How important is computer security in your domain?


Comments:

B. Security requirements, design, and regulations

7. What types of security components do you use/manifacture/deploy?(Multiple selections are possible)

2 Firewall

2 Application White-listing

2 Intrusion Detection System

2 Intrusion Prevention System

2 Forensic Analysis Tool

2 Other:

Comments:

8. Rank these limitations to security in order of importance from 1 (Highimportance) to 6 (Low importance).

� Costs are a big factor limiting the amount of research and spending on security.

� Time constraints. No time to investigate all potential security issues andsolutions.

� Off-the-shelf security technology are not suitable for CIs and their proprietarysoftware.

� The underlying hardware used in most CI cannot handle modern securityfeatures.

� Regulations/laws do not allow all security features.

� There is no incentive to secure every aspect of CIs.

� Other:

Comments:


C. Testing and methodologies

9. * How important is the correct functioning of security components inyour system?

(little) #—#—#—# (a lot)

Comments:

10. * How is the access to the systems’ information and specification con-trolled? (Multiple selections are possible)

2 Only those who have a need-to-know can access such information

2 For some specific aspects a very small set of people has access to such infor-mation

2 Specific area of the company terrain are only accessible by authorized personel

2 Only the manufacters/suppliers have the full specification of the components

2 Open source software is used

2 Also internally, the security tests can mostly be done as black-box tests

2 Other:

Comments:


11. What kind of security tests are done? (Multiple selections are possible)

2 Stress tests and Denial-of-Service tests

2 Penetration tests

2 Security code review

2 Formale code verification

2 Other:

Comments:

12. How often are these tests performed? (Multiple selections are possible)

2 At design time


3. Requirements

2 During development

2 During the setup

2 Regularly during operation

2 When some changes occur

2 Other:

Comments:

13. Are security tests mainly black-box (system’s specifications unknown)or white-box (full disclosure of information to pentesters)?

# Black-box

# White-Box

# Unknown

Comments:

14. * Is it possible to test the system remotely (passing through publicnetworks)?

# Yes

# No

# Unknown

Comments:

15. * Is the security testing outsourced, i.e. done by externals?

# No, everything is tested in-house.

# No, generally everything is done in-house but in some cases external securitytesters are included.

# Yes, but only specific elements are tested by externals.

# Yes, the whole system is tested for security by externals.

# Unknown

# Other:

Comments:



16. Is the testing performed within company premises?

# Yes, always.

# If outsourced, it can be performed also outside company premises.

# Unknown

# Other:

Comments:

17. For web servers security/penetration testing methodologies and toolsexist.Are there any methods/procedures/standards used for testing CI se-curity?

# No, I am not aware of any general procedures/standards.

# No, there are no standards but some methodologies specifically designed foreach CI exist.

# Yes, internally a set of documented procedures exist which are followed foreach CI.

# Yes, published/standardized methods are used for testing wherever possible.

# Unknown

# Other:

Comments:

18. Can you mention some methods used?

19. What are the procedures for security testing CIs? How is the securitytested? (Multiple selections are possible)

2 A security test is done on the whole “live” system.


3. Requirements

2 A security test is done when the system is offline.

2 A security test is done on a backup system.

2 Different components are tested separately

2 Unknown

2 Other:

Comments:

20. The security tests are done by... (Multiple selections are possible)

2 The team working on the component(s).

2 An internal security team.

2 An external security team.

2 Unknown

2 Other:

Comments:

21. Are there any CI-specific tools/software for the testing? (Multipleselections are possible)

2 No, I am not aware of any CI-specific tool.

2 Some simple tools are developed when necessary.

2 Yes, existing software tools are used for testing specific components.

2 Yes, we developed our own tools/software for testing the security of the ICS.

2 Unknown

Comments:

22. Referring to previous question, can you mention any of such tools/soft-ware?



23. Is there a need for a (standard) CI penetration testing methodology?

# No. There is no incentive to do thorough security testing.

# A little. The methodologies/documented procedures will not be used. We do aflexible security testing strategy that is decided upon for each individual case.

# Yes. General guideline methodologies are good, but there is no need for morespecific methodologies.

# Yes. A CI-specific and comprehensive methodology would make security test-ing more organized/ repeatable/ comparable.

# Unknown

Comments:

24. This CI penetration testing methodology should... (Multiple selectionsare possible)

2 include a guideline for testing the whole CI.

2 include a guideline for testing specific components.

2 include general guidelines about each testing aspect.

2 include very detailed information about each testing aspect.

2 suggest software/tools to be used.

2 state how to document the results.

2 Other:

Comments:

25. Are there any areas/aspects/topics that need special attention?


3. Requirements

D. Research

26. Should more research be done in the area of CI security (either atuniversity or at companies)?

# No, there is no incentive to do thorough security testing.

# No, there is already a lot being done (also in non-CI areas that can be appliedto CI).

# Yes, more time should be spent in studying CIs’ security.

# Other:

Comments:

27. Additional comments:

E. Follow up

28. Would you agree to be contacted for a more in depth discussion?

# Yes

# No

29. If yes, please provide us your email address:

Thanks for your time. In case you provided your email address, we will contact you toorganize a 1 hour phone interview.

3.3. Further analyses

At the moment of this writing (April 2013) the distribution and filling out of the ques-tionnaire is still in progress. Due to the limited feedbacks we collected until now, it is



difficult to generalize the results. However we are starting to see some trends and com-mon believes in the answers. Interviewed people belong to Academia, SCADA vendorsand utilities, consulting companies, etc. This is one of the main reasons why we proposea two step approach to D5.1 were we provide a preliminary proposal now that is laterextended and validated based on further input. We still provide a preliminary summaryof the found requirements in this section.

As we expected, Critical Infrastructures and Security seem to be two inseparableconcepts. All the participants to the survey agreed on considering computer securitya priority in implementing and managing Critical Infrastructures. However there aresome limitations in deploying security. Most of the interviewed experts identified costsas the biggest obstacle to properly ensure security. Costs are always a big factor limitingthe amount of research and spending on security. Almost equally important as costs aretime constraints. Several people complain about the limited time available to investigateall potential security issues and solutions. Ranked a bit lower, Off-the-Shelf securitytechnologies in combination with unusual hardware used in SCADA seem issues easierto resolve. On one hand, Off-the-Shelf tools and techniques can be customized and usedon CIs and their proprietary software. On the other hand, CI’s underlying software,even outdated, can handle modern security features. Finally, participants agreed on thefact that laws and regulations already in place do not impede operators to deploy newsecurity features to their infrastructures.

It can be deduced that a penetration testing technique and, more in general, an as-sessment methodology for Critical Infrastructures has to focus on reducing cost and beingtime efficient. Standard methodologies have to be reduced to a minimum sequence ofactivities in order to save money and time. Automation of testing steps will play animportant role. Penetration testers and auditors have to cut all disposable processesfrom their work and maximize the throughput. At the same time, external testers mayhave a clearer time and cost budget they can plan with.

According to the results of our interviews there are no preferences in choosing a spe-cific kind of security mechanisms with respect to the others. Firewalls, ApplicationWhite-Listing, Intrusion Detection and Prevention Systems and also Forensic Analysistools are widely deployed and used in Critical Infrastructures. So a penetration testingmethodology should not focus on one of them but include them all.

The same is true for the different kinds of security test. In the questionnaire weproposed four types of tests asking which of these are already implemented in CriticalInfrastructures. Our experts confirm that Stress tests and Denial-of-Service tests, Pen-etration tests, Security code reviews, and Formal code verification are quite known andused in CI field. For this reason, a security assessment can freely exploit one or more ofthem depending on the purpose.


3. Requirements

All tests are so far mostly done during CI development and setup. According toour results, only a small percentage of the survey participants considers a “live” systemsuitable for a penetration test. However, situations in which the system is off-line formaintenance seem suitable to run specific tests. The reason behind these considerationsrelies on the concern that a test may affect the proper functioning of Critical Infrastruc-ture and endanger both physical processes, people, and company business. Therefore,security assessments must not have an impact on live systems. Every action must beevaluated beforehand with respect to this risk. In any case, security operators have to beready to face problematic or dangerous situations that can be caused by a penetrationtest.

Questions about strategies and constraints on penetration testing reveal that there isno common agreement on the issue. First of all, Critical Infrastructures use both White-Box and Black-box penetration testing depending on the purpose of the test. Together,the two techniques contribute to perform a comprehensive assessment that focuses bothon vulnerabilities and defense reaction capabilities. However, some experts exclude oneor the other for different reasons. Time constraints and potential dangerous impacts aresome of them. Remotely accessing the network is another big issue. Not all the operatorsare available to open their systems to Wide Area Networks (WAN) or to the Internet.On the other hand, some of them agree on passing through public networks to bettersimulate real attacks. Finally, some operators agree on outsourcing security testing andgiving access to the system to third parties. These tests are usually performed withincompany premises. In any case, outsourced tests cannot substitute internal testing.However, almost all the interviewed persons agreed that an outsourced test is alwaysspecific for a component or a subsystem but never general and comprehensive of thewhole Critical Infrastructure.

The majority of respondents is not aware of any specific penetration testing method-ology for Critical Infrastructure. Some experts suggested that (non-public) customizedmethodologies for specific tests exist but we can safely conclude that no comprehensivesecurity assessment methodology has been developed and published yet. We know fromthe results that several Off-the-Shelf tools and software are used to test Critical Infras-tructures. Some of the experts provide examples. Metasploit [14], Nessus [1], Achilles[15, 16], Mu Dynamics products [17] but also standard IT fuzzers and code checkingtools are used in the CI context. In several cases, operators and vendors developed theirown tools or customized standard one for specific purposes.

At the end of the questionnaire we evaluated how a new assessment methodologyspecifically tuned on Critical Infrastructure would be received within the community.Almost all the interviewed agreed on the need for a standard CI penetration testingmethodology. Such an opinion is however divided in two different connotations. On the



one hand, half of the experts say that general guidelines methodologies are good, butthere is no need for more specific methodologies. On the other hand, the others ask for aCI-specific and comprehensive methodology that would make security testing more orga-nized/repeatable/comparable. Again, the requests about what a methodology should doare various and different among each other. Almost all the responses want the method-ology to suggest specific software/tools to be used. The experts also emphasize the needfor a comprehensive evaluation capable to provide general guidelines about each testingaspect. Moreover, it is important to define how to document testing results in order tohave a common background useful for comparisons. Assessing specific components withrespect to a general evaluation of the infrastructure seems a priority. This is probablydue to the difficulty of analyzing results related to large and, sometimes, geographicallydistributed infrastructures. It is worth noting that almost no one asks for very detailedinformation about each testing aspect. This is related to the difficulties that a detailedmethodology will have to face with respect to already in place policies and proceduresinside specific Critical Infrastructures.

Thanks to the questionnaire and interview feedback, we have a first insight on whatoperators and vendors want from a CI assessment methodology. The sometimes varyinganswers reflects the real world situation where there is no one-size-fits-all solution. Fromsystem to system and from company to company specific needs lead to the developmentand customization several different testing methods. The idea behind this deliverable isto collect such different information and create a general methodology based on com-mon elements among already existing methods. At the same time, the new CRISALISpenetration testing methodology needs to be flexible enough to be adaptable to differentscenarios and environments.


4. Methodology

4.1. Overview

In this section we focus on formalizing and merging all the concepts described in theprevious two sections. The output of this analysis will be a new methodology for asecurity assessment methodology for Critical Infrastructures. We need to note that thecurrent proposal should only be seen as a draft and that the concluded survey togetherwith experiences with applying our approaches to testbeds available in CRISALIS willlead to a refinement and extension of this methodology in the second version of thisdocument.

Our work has two different starting points. On one hand we have the inputs providedby the questionnaire and the interviews. This is a necessary contribution for differentreasons. First of all, experts in the field (e.g. SCADA vendors, utilities, security con-sultants, etc. ) have provided their expertise that we are building on. Thanks to theirknowledge we were able to identify problems and concerns linked to the application do-main and constraints in an industrial environment. Moreover, they allow us tailor ormethodology to their actual needs. The interviewees explained how assessments andpenetration tests are performed today in CIs and what they consider is still missing inthese evaluations. Our CRISALIS penetration testing methodology addresses these is-sues and provides a generalized approach capable of taking the best of specific assessmentmethods already used, unifying differences and overcoming limitations.

On the other hand, we have examined a number of relevant security assessmentmethodologies adopted from standard IT. OSSTMM, NIST SP800-115 and ISSAF arewell known and extensively used. However, we see some shortcomings. E.g., several ofthe steps may not always be needed in the Critical Infrastructure field. Our idea of asecurity assessment methodology for Critical Infrastructure is a process that adopts theoverall structure, the terminology, and certain individual elements or steps from thesestandard methodologies. However, at the same time, it leaves aside details regarding ITelements that are of secondary importance (or are completely missing) in the industrialfield. Even if Industrial Control Systems and standard IT networks are merging more andmore, assessing Critical Infrastructure must include also verifying specific componentsand properties of such heterogeneous systems such as PLC. The scope can therefore be

50

4.2. Workflow

narrowed but some additional steps must also be included.

Our work maps the needs identified by the experts on standard security methodolo-gies. We do not want to create a new standard from scratch but rethink the existingones. The result will be a lightwheight security assessment methodology involving apenetration testing methodology tuned to industry components and constraints. Theuse of a common schema to evaluate Critical Infrastructure will allow operators, ven-dors and security consultants to share information and assessments in an easier way andwill make results more meaningful. A uniform approach will allow to easily spot majorproblems and threats inside Critical Infrastructure and will eventually contribute to ageneral improvement of the security of such systems.

4.2. Workflow

4.2.1. Main Workflow

As introduced in the previous paragraph, we conceptually split our methodology in twomajor parts:

� General Assessment: it covers all the aspects of a security assessment. It in-volves three sub-phases: a Pre-Assessment in which a company identifies the goalsof the assessment and solves any bureaucratic and legal issue regarding the follow-ing operations; the actual Practical Assessment in which auditors and pentestersperform practical tests on the assets; and the Post-Assessment phase in which thecompany evaluates the results and decides about countermeasures and informationdisclosure.

� Practical Assessment: it represents the intermediate part of the General As-sessment. It focuses on practical tests performed on Critical Infrastructures. ThePractical Assessment is also divided in three different sub-phases: the Preparationin which pentesters discuss technical details with system operators and preparetools and strategies to be used during the test; the Testing where all the tests areperformed; and the Analysis in which the results of the tests are documented anddetailed.

Figure 4.1 shows the methodology workflow.

There are two different contact points between the two methodology’s parts. Theformer links the Pre-Assessment phase to the Preparation one while the latter links theAnalysis phase to the Post-Assessment one. In both the cases there is a substantial


4. Methodology

Figure 4.1.: Critical Infrastructure Methodology

movement of information flowing from a phase to the other. We will denote these twoactivities as Information Flows one (’1’) and two (’2’).

Finally, there are two possible loops in the workflow. The first feedback is given by theAnalysis phase. At the end of the practical assessment it is possible to restart it withoutpass to the Post-Assessment. This situation describes a case in which the requirementsidentified in the General Assessment do not match with the actual testing. The secondloop concentrates on the Testing phase and will be described in 4.2.3.

Before discussing each working phase in details, it is worth noting that the reason toseparate the methodology in two different parts is twofold. The first reason concerns theusability of the methodology. Such schemes allows operators to perform one of the twoparts independently. Let us imagine a situation in which the procedures described forthe Practical Assessment are not compliant with company policies or they are simplynot suitable for company’s purposes. In this case, operators can exploit the GeneralAssessment scheme to perform all the accessory operations and substitute the PracticalAssessment with a different model in line with their expectations. Thanks to the pro-posed methodology they will have just to take care about the Information Flows andmodify them properly. In the same way, operators can use just the Practical Assessmentwith the condition to provide the right input and to modify the output by taking into


4.2. Workflow

account requests provided by a different security assessment.

The second reason behind the proposed scheme regards the maintainability. CriticalInfrastructures have changed a lot since the ’70s and they are likely going to change evenmore in the future. Moreover, legislations and company policies also change over time.The possibility to substitute a part of the methodology without altering the rest is adefinite advantage in terms of cost and time. In case of such modifications, operators willhave to modify the Information Flows to link the new part to the rest of the methodology.

4.2.2. Workflow phases

In what follows we describe the five phases and the two Information Flows more indetails:

� Pre-Assessment: the Pre-Assessment is the starting point of our security eval-uation methodology. In this phase the company opens the discussion about thesecurity assessment and explains its motivations and its goals. This phase shouldinvolve at least three different actors: the Chief Security Officer (CSO) or who-ever is representative for this function and may decide to approve the starting ofassessment operations; the Chief Financial Officer (CFO) or whoever is represen-tative for this function and may decide on the budget allocated for the assessmentoperations and about acceptable risks; and ICS and SCADA operators that will beresponsible to contribute to the discussion with the technical point of view. Thediscussion deals with several topics. First of all, the actors have to decide aboutsecurity assessment’s purposes. The main goal and the motivations behind thefollowing operations have to be stated clearly. What follows is the identificationof company assets. The actors have to focus on one or more targets accordingto the proposed objectives. Critical Infrastructures are complex systems and, forthis reason, it is unlikely to have always a comprehensive assessment. Most of thetime, specific sections or processes of the Critical Infrastructure will be checked andanalyzed. An additional and very important step is a detailed risk analysis thatclearly identifies activities endangering the safety of the system and its environ-ment. This needs to be documented as well as an agreement to either skip thosesteps or perform them with specific safety guards. During the Pre-Assessment,involved actors have to agree on the time plan and decide on how to distributeresources to the five phases of the methodology. Finally they have to investigateand ensure that legal and contract obligations regarding assessment operations aremet. This last step overlaps with the Control Assessment proposed by ISSAF andalready discussed in 2.3.1.


4. Methodology

� Information Flow 1 : at the end of the Pre-Assessment phase its participantshave to prepare a document summarizing all the results of the discussion. TheInformation Flow describes the work the operators will do for translating suchoutcomes in practical directives for the pentesters. For example, the assets iden-tified in the previous phase will become systems and components to be tested. Inthe same way, the main goal will be divided in several sub-goals which will bemapped into specific tests. Such information will be the input of the Preparationphase.

� Preparation: the Preparation is the first phase of the Practical Assessment. Thisphase involves two actors: ICS and SCADA operators that report the outcomes ofthe Pre-Assessment phase; and the pentesters which are in charge of performingnecessary tests. The two actors can can be from the same organization if thepurposes or the constraints of the security assessment respectively allow or do notpermit the presence of third parties (e.g. external companies, trained pentesters,etc.). Operators and pentesters have to discuss several issues. First of all, they willdecide on what kind of pentesting schema to follow. They will decide on the typeof penetration testing as proposed by OSSTMM in 2.1.2 (e.g. Blind testing, Greytesting, etc.). Moreover, they will choose on the timing of the attack (e.g. workinghours, weekends, etc.). Finally, they will decide whether to attack the live system,or a backup, or performing all the tests when the system is off-line. The attackimpact is one of the main concerns that has to be faced during the Preparation.Critical Infrastructures cannot be put in danger by a penetration testing even ifthe reason to perform the analysis is verifying that possibility. During this phaseoperators should decide and implement specific safeguards to put in place duringtests and prepare an emergency plan with countermeasure to take in case one ofthe security tests succeed in penetrating and corrupting the system. This planhas to be ready also in case of defensive black box testing. In that situationoperators participating to the test will not be aware of the plan unless a problemoccurs. In this phase operators have to verify if all the objectives defined duringthe Pre-Assessment are achievable. Otherwise they should motivate any problemthat represents an obstacle to the test. The final step of this phase is the setup ofthe environment and of all the resources needed to the pentesting activity.

� Testing: the Testing phase implements the security tests that have been discussedduring the Preparation. Its duration depends on different factors such as: accuracyof the tests (e.g. comprehensive tests vs. vulnerability identification), pentesterskills, allocated budget, etc. During the entire phase pentesters have to log all


4.2. Workflow

the performed operations and related discoveries. Such report will be the inputof the Analysis phase. Security operators responsible for the emergency plan willfollow the operations from a different point of view. They will check the presenceof penetration side effects on the Critical Infrastructure that the pentesters arenot aware off. Such information will be integrated to the pentest log at the endof the Testing phase. It is worth noting that the testing plan arranged in thePreparation phase can change accordingly to the evolution of the tests. Unexpectedobstacles can interfere with the penetration test and undermine the feasibility ofsome operations. Such situations have to be also detailed in the log. In case atest shows severe vulnerabilities that can endanger the Critical Infrastructure, thesafeguards and emergency plan should be triggered and prevent real damage. Theeffect of such an event is the interruption of the methodology workflow and theimmediate feedback to the Chief Security Officer. The “level of severity” neededto trigger the emergency plan has to be identified in the Preparation phase. Thetesting phase will be extensively discussed in section 4.2.3.

� Analysis: the Analysis phase concludes the Practical Assessment. As in thePreparation phase, pentesters and operators are the two actors involved. Theformer will report about the findings of the Testing phase and will propose arisk rating of the security to the Critical Infrastructure. The latter will criticallyreview the testing, contextualizing the problems and and justifying any design ormaintenance choices. Moreover, operators have to perform two important tasks:check if the status of the system has returned to normal after the testing session;and apply fixes to minor and easy-to-fix problems and vulnerabilities. At the end ofthe Testing phase, the pentesters will have to undo permanent modifications to thesystem, like removing any file or software installed inside the Critical Infrastructureor unpluging any machine used for pentesting. During the Analysis, operators willcheck the correctness of such activity and will run some integrity checks to verifythat the Infrastructure is brought back to a safe state. It is probable that problemssuch as misconfigurations and other simple activities from the penetration inducesome errors into the system. Operators should fix such issues without reportingthem to the next stage of the methodology. In this sense, the Analysis phase isalso a filter identify the more severe security problems and what types of problemsa later security deployment may create.

� Information Flow 2 : at the end of the Analysis phase operators and pentesterswill prepare a document that, starting from the report of the Testing phase, sum-marizes findings and security issues. The format of the documentation can be based


4. Methodology

on the the ones proposed by ISSAF in its Final Reporting 2.3.2 and partially alsoon the final requirements of the OSSTMM 2.1.2. Some information needed to thedocument describing the Analysis phase are:

– Date and time of test

– Duration of test

– Auditors and analyst involved

– Tools that have been used

– Test type

– Scope of test

– Comprehensive list of performed tests

– Knowledge of which tests have been completed, not completed, or only par-tially completed, and to what extent

– Test error margins

– Vulnerability discovered/exploited with recommendations on how to solveproblems found

– Unknown anomalies

Operators will have to motivate and justify such documentation during the Post-Assessment phase.

� Post-Assessment: the Post-Assessment closes our security assessment method-ology. With respect to the Pre-Assessment phase the presence of the Chief Fi-nancial Officer is no longer needed as any decision regarding new operations willbe taken afterwards. Therefore, in this phase, the discussion involves ICS andSCADA operators and the Chief Security Officer or whoever represents this rolein the company. Operators will explain the steps performed during the PracticalAssessment. Pentester can also participate to the meeting. Their task will beto present their findings focusing on major problems encountered. Operators willalso propose countermeasures agreed on the basis of the discussion in the Analysisphase with the pentesters. The CSO will evaluate the proposed countermeasuresdeciding on what actions to take to solve the most pressing issues. It is worthnoting that, during the Post-Assessment phase, participants will also discuss thepossible impact of vulnerability disclosures. The output of this phase is the finalresult of the security assessment methodology. Such documentation is both anhigh-level overview about company’s vulnerabilities, possible mitigations, limita-tions in deploying security and relation to security legislation or policies on the


4.2. Workflow

one hand and on the other hand a low-level overview describing specific problemsand vulnerabilities. The conclusions of the assessment match the list proposed bythe NIST in 2.2.2 (Insufficient patch management, Lack of security baselines, In-adequate incident response procedures, etc.). The output of the Post-Assessmentmethodology can be used for following security assessments (e.g. to identify solvedand still in place problems and vulnerabilities) or as a starting point for differentkinds of assessment (e.g. risk assessment introduced in CRISALIS deliverable 2.2).

Figure 4.2 shows a summary of the concepts discussed above. We will now discuss thepenetration testing in more detail.

Figure 4.2.: Critical Infrastructure Methodology Detailed Schema

4.2.3. Penetration Testing Steps

The penetration testing process is the core of the assessment methodology. Duringthe Testing phase, pentesters go through five operational steps: Information Gathering,Network Mapping, Vulnerability Identification, Penetration, and Maintaining Control.


4. Methodology

Each step groups a set of operations necessary for the next step. We also envisiona circular representation of the testing steps as the penetration testing process is notnecessarily “one-time”. A successful attack and the consequent control of a part of thesystem by the pentesters could open new attack vectors. In this case, pentesters cantraverse again through the five pentesting steps exploiting new targets or attack vectors.The process ends when all the prepared attack scenarios as well as those discoveredduring the activity have been tried and evaluated.

The specifications of the five penetration testing steps are based on the schema pro-posed by ISSAF in 2.3.2 as we consider this the most appropriate and comprehensiveapproach as far as the actual penetration testing is concerned. In what follows wedescribe each step in detail:

� Information Gathering: the Information Gathering step focuses on systems forcollecting and analyzing information regarding the Critical Infrastructure. Thisinformation concerns several elements such as working personnel, company policies,used hardware/software, etc.. The Information Gathering involves both technicaland non-technical methods. Depending on the type of testing (Black Box vs.White Box), pentesters can have some initial information at their disposal thathelps them to understand the environment they are going to attack. It is unlikelyto find useful information about the Critical Infrastructure internal functioningon the Internet. However, the knowledge of the working staff and their skills andbackgrounds can give some ideas on what pentesters will have to face once theattack has started. In case of a White Box penetration testing the InformationGathering is important to quickly identify points that will be most likely vulnerableand focus on them.

� Network Mapping: the Network Mapping step directly follows the InformationGathering and contributes to acquire more information about the Critical Infras-tructure. In this case, pentesters concentrate only on hardware and software. Themain target is to identify what devices are working inside the Critical Infrastruc-ture, identify communication patterns the network topology. Different tools canbe used during this activities (e.g., sniffers, fingerprinters, etc.). Approaches to fin-gerprint devices and map networks are investigated in WP4 of CRISALIS and arepresented in deliverables 4.1 and 4.2.. Such tools can be highly useful to automatesome work in this step and thus save time and costs. During this step pentesterswill perform the following operations:

– Topology discovery and finding live hosts


4.2. Workflow

– Differentiate from network components (e.g. switch, firewalls, etc. ), ICSdevices (PLC, RTU, etc.) and standard computers (servers, laptops, etc.).

– Port and Service scanning

– Operating System and Services Fingerprinting

– Critical Services Identification

The Network Mapping step will allow pentesters to confirm or discard attack hy-potheses formulated during the Preparation phase or the Information Gatheringstep.

� Vulnerability Identification: once one or more targets are selected, pentesterswill perform a vulnerability scanning. The most commonly used tools for vulnera-bility identification in Critical Infrastructure are Achilles [15] [16] and Nessus [1].The list of the activities of this step comprises but is not limited to:

– Perform vulnerability scans

– Enumerate discovered vulnerabilities

– Perform vulnerabilities verification (e.g. trying to eliminate false-positive)

– Estimate possible impact of exploiting found vulnerabilities and verify thatattack can be performed safely

– Identify suitable attack vectors for exploitation

� Penetration: during the Penetration step, pentesters try to gain access and ob-tain control of Critical Infrastructure’s components. This is the most dangerousoperation of the Testing phase as the attackers try to subvert normal operationsof the system or a part of it. The Metasploit Framework [14] is one of the mostwidely adopted tool to exploit software vulnerabilities. In its professional ver-sion, the framework also provides several exploits against Programmable LogicControllers (PLC). However, most of the times, pentesters will resort to developcustomized scripts and tools to exploit ICS-specific vulnerabilities. This is the caseof the fuzzers developed within the CRISALIS project and presented in deliver-able 5.2. What follows is a list of operations performed by pentesters during thePenetration step:

– Find or develop suitable code/scripts/tools to use for the attack

– Test developed code/scripts/tools before using them on the Critical Infras-tructure (to avoid problem during pentesting and unwanted effect on thewhole system)


4. Methodology

– Use code/scripts/tools against the Critical Infrastructure

– Verify or disprove the existence of vulnerabilities

� Maintaining Control: once in control of a machine, attackers will verify whatinformation can be retrieved from the machine and if they can use the compromisedcomputer to find new targets on the network. Pentesters will search for valuabledata like:

– Passwords

– Running processes

– Information regarding the physical process under control

– Generic contacts (e.g. emails)

If the compromised machine is at the edge of the network it may also link todifferent networks. Pentesters have to evaluate this possibility and, if this is thecase, restart the pentesting process against the new target. Attackers have alwaysto verify the possibility to maintain access to the machine for future activitiesby installing malicious code (e.g., root-kits and backdoors). This guarantees tomaintain control even if the exploited vulnerability is patched. Whether this shouldbe conducted needs to be decided in the planing phase. Finally, pentesters haveto cover tracks of the intrusion. Such operations are usually performed:

– Hiding modified files and tools

– Cleaning logs

– Restoring integrity checks

At the end of this step the Testing restarts on a new target or ends up lettingpentesters proceed with the Analysis phase.

Figure 4.3 proposes a comprehensive schema of the assessment methodology plus thechain of the five penetration testing steps.

4.3. Final Remarks

The presented CRISALIS penetration testing approach represent a first draft towards asecurity assessment methodology for Critical Infrastructures. To the best of our knowl-edge, this work is the first attempt to formalize Critical Infrastructure assessments andpentestings. With our proposal we tried to incorporate requests and advices given byCritical Infrastructure experts with existing standard methodologies. The result is a


4.3. Final Remarks

Figure 4.3.: Critical Infrastructure Methodology plus Penetration Testing Steps


4. Methodology

light-weight but comprehensive list of working phases linked by specific information anddocumentation exchanges that should lead pentesters through a fast, cost-saving andeffective evaluation of a system.

Our methodology differentiates among two logical parts. The first, called “GeneralAssessment”, concentrates on the administrative process that a company must followto organize a security assessment. The second, called “Practical Assessment”, regardsthe technical preparation of the evaluation process and the consequent analysis. Finally,the process shown in 4.2.3 describes the actual penetration testing. In this phase wediscussed the steps pentesters have to follow to check and evaluate a Critical Infrastruc-ture. As described in the introduction of this chapter, the use of separated but connectedlogical components improve both usability and maintainability of the methodology.

The proposed methodology is not refined in arbitrary level of detail and yet needs tobe tested in a Critical Infrastructure. This will be done once the survey and interviewresults are fully available as this will also provide a detailed understanding of the levelof detail that is actually required.

In the meantime, we have conducted some initial penetration testing experiments inone of the CRISALIS testbeds where one new vulnerability was found. These tests alsohelped to shape and refine this methodology as we were able to identify importance andfeasibility of different steps. It also highlighted that there are insufficient structures toreport newly found vulnerabilities of incidents to manufacturers and users. This shouldbe addressed in a second version of this deliverable.


5. Conclusion

In this deliverable, we have started with motivating the need and benefits for a method-ology for security penetration testing for Critical Infrastructures and Industrial ControlSystems. Benefits include time and cost savings for penetration testers, easier com-parison of results, and better confidence in completeness and correctness of results. Weanalyzed a series of existing methodologies regarding their applicability in the CRISALIScontext. Based also on the preliminary feedback from a survey and interviews we haveconducted, we identified requirements and the need to provide an adapted and revisedapproach. While we make a first proposal for such an approach already in this versionof the deliverable, we suggest to provide a second version of the document by the end ofY2, as this will allow us to finish the survey, use this to provide an updated and refinedmethodology, and finally also include results from own penetration testing experimentsin our testbeds.

63

Bibliography

[1] R. Deraison, H. Meer, and C. Van Der Walt. Nessus network auditing. SyngressMedia Incorporated, 2004.

[2] Wikipedia. White-box testing — Wikipedia, the free encyclopedia, 2004. [Online;accessed 14-January-2013].

[3] R. Patton. Software testing. Sams, 2005.

[4] Wikipedia. Black-box testing — Wikipedia, the free encyclopedia, 2003. [Online;accessed 14-January-2013].

[5] Wikipedia. Gray-box testing — Wikipedia, the free encyclopedia, 2006. [Online;accessed 14-January-2013].

[6] M. Prandini and M. Ramilli. Towards a practical and effective security testingmethodology. In Computers and Communications (ISCC), 2010 IEEE Symposiumon, pages 320–325. IEEE, 2010.

[7] D.P. Duggan, M. Berg, J. Dillinger, and J. Stamp. Penetration testing of industrialcontrol systems. Sandia National Laboratories, 2005.

[8] P Herzog. Osstmm 3–the open source security testing methodology manual.barcelona, espana: Isecom, 2010.

[9] Karen Scarfone, Murugiah Souppaya, Amanda Cody, and Angela Orebaugh. Nistspecial publication 800-115: Technical guide to information security testing andassessment, 2008.

[10] Creighton T Hager and Scott F MidKiff. An analysis of bluetooth security vulner-abilities. In Wireless Communications and Networking, 2003. WCNC 2003. 2003IEEE, volume 3, pages 1825–1831. IEEE, 2003.

[11] Creighton T Hager and Scott F Midkiff. Demonstrating vulnerabilities in bluetoothsecurity. In Global Telecommunications Conference, 2003. GLOBECOM’03. IEEE,volume 3, pages 1420–1424. IEEE, 2003.

64

Bibliography

[12] Balwant Rathore, Mark Brunner, Miguel Dilaj, Omar Herrera, Piero Brunati,Rama K Subramaniam, Subash Raman, and Umesh Chavan. Issaf 0.2.1 - infor-mation systems security assessment framework, 2006.

[13] Marco Caselli and Frank Kargl. Critical infrastructure security & penetration test-ing questionnaire. https://docs.google.com/spreadsheet/viewform?formkey=

dG15cUV5ck9HWlg1eTVuVk51ZnNYb1E6MQ/, 2012.

[14] LLC Metasploit. The metasploit framework, 2007.

[15] Achilles test platform. http://www.wurldtech.com/product_services/

discover_analyze/achilles_test_platform/.

[16] Achilles test software. http://www.wurldtech.com/product_services/

discover_analyze/achilles_test_software/.

[17] Mu dynamics software. http://www.mudynamics.com/products/.


https://docs.google.com/spreadsheet/viewform?formkey=dG15cUV5ck9HWlg1eTVuVk51ZnNYb1E6MQ/

https://docs.google.com/spreadsheet/viewform?formkey=dG15cUV5ck9HWlg1eTVuVk51ZnNYb1E6MQ/

http://www.wurldtech.com/product_services/discover_analyze/achilles_test_platform/

http://www.wurldtech.com/product_services/discover_analyze/achilles_test_platform/

http://www.wurldtech.com/product_services/discover_analyze/achilles_test_software/

http://www.wurldtech.com/product_services/discover_analyze/achilles_test_software/

http://www.mudynamics.com/products/

Documents

D5.1 Security Testing Methodology - CRISALIS Projectcrisalis-project.eu/sites/crisalis-project.eu/files/crisalis... · D5.1 Security Testing Methodology Contract No. FP7-SEC-285477-CRISALIS