Embed Size (px)
Citation preview
12/6/2010
Design Document | Tony Gedwillo – James Parrott – David Ryan
SDMAY11-11 CYBER SECURITY OF SCADA SYSTEMS TEST BED
Design Document
Team Members: Tony Gedwillo – James Parrott – David Ryan
Faculty Advisor: Dr. Manimaran Govindarasu
SDMAY11-11 1
Table of Contents List of Figures ................................................................................................................................................ 3
Executive Summary ....................................................................................................................................... 4
Acknowledgement ........................................................................................................................................ 4
Problem Statement ....................................................................................................................................... 4
General Problem Statement ..................................................................................................................... 4
General Solution Approach ....................................................................................................................... 5
Operating Environment ................................................................................................................................ 6
Intended Users and Uses .............................................................................................................................. 6
Intended Users .......................................................................................................................................... 6
Intended Uses ........................................................................................................................................... 6
Assumptions and Limitations ........................................................................................................................ 6
Assumptions List ....................................................................................................................................... 6
Limitations List .......................................................................................................................................... 6
Expected End Product and Other Deliverables ............................................................................................. 6
Approach Used .............................................................................................................................................. 7
Design objectives ...................................................................................................................................... 7
Functional Requirements .......................................................................................................................... 7
Virtualization ......................................................................................................................................... 7
Power System Simulation and Integration ........................................................................................... 7
Cyber Security Assessment ................................................................................................................... 8
Design Constraints .................................................................................................................................... 8
Technical approach considerations and results ........................................................................................ 9
Virtualization Approach ........................................................................................................................ 9
Power System Simulation and Integration Approach ......................................................................... 11
Cyber Attack/Security Approach ......................................................................................................... 11
Testing approach considerations ............................................................................................................ 13
Virtualization Testing .......................................................................................................................... 13
Power System Simulation and Integration Testing ............................................................................. 14
Cyber Security Testing......................................................................................................................... 14
Recommendations regarding project continuation or modification ...................................................... 15
Detailed Design ........................................................................................................................................... 15
SDMAY11-11 2
Virtualization: .......................................................................................................................................... 15
Overview ............................................................................................................................................. 15
Power Flow Simulation and Integration ................................................................................................. 16
Cyber Security Vulnerability Assessment ................................................................................................ 19
Project Team Information ........................................................................................................................... 21
Faculty Advisor Information .................................................................................................................... 21
Team Information ................................................................................................................................... 21
Closing Summary ......................................................................................................................................... 21
SDMAY11-11 3
List of Figures Figure 1: Design Cycle Diagram ..................................................................................................................... 5
Figure 2: Sample Nessus Workstation Report ............................................................................................ 12
Figure 3: Sample Nessus Vulnerability List ................................................................................................. 13
Figure 4: System Diagram ........................................................................................................................... 16
Figure 5: One-Line Diagram from PowerFactory ........................................................................................ 17
Figure 6: Using Spectrum Power TG to close a relay .................................................................................. 18
Figure 7: Conceptualization of our testbed's software communicaiton .................................................... 19
SDMAY11-11 4
Executive Summary Supervisory Control and Data Acquisition (SCADA) systems are the nervous systems for the body of our country’s infrastructure. This body includes many systems that are vital to the function of our society: power, water, natural gas, oil, and road traffic systems—among many others. However, the nervous systems (SCADA systems) that control our infrastructure are currently vulnerable to cyber-attack. “Since the mid-1990’s, security experts have become increasingly concerned about the threat of malicious cyber-attacks on the vital supervisory control and data acquisition (SCADA) systems used to monitor and manage our energy systems. Most SCADA system designs did not anticipate the security threats posed by today’s reliance on common software and operating systems, public telecommunication networks, and the Internet.” With the critical infrastructure of the SCADA systems and the security threats on these systems, it is important to research ways to correct potential security vulnerabilities. A SCADA test bed will be used for this research. This project will expand on the initial test bed created last year and make it more suitable for real-life scenarios and cyber security attacks. The previous senior design team created the initial SCADA test bed. This test bed included 2 Control Centers, 2 RTUs, 2 Relays, 3 SCALANCEs for encrypted communication, a web server, a DTS, and a light board for demonstrating when a relay trips or is closed. The previous team also tested basic cyber-attacks against the system. They were able to demonstrate a basic man-in-the-middle attack that would disrupt commands sent by the control center. The initial test bed was a great start and this year’s senior design team will improve on the test bed. The goals of this year’s senior design team are to expand the test bed to more nodes, integrate power flow analysis and test more advanced attacks. The basic approach for these goals is to use virtualization software to expand the test bed’s nodes, use power flow software for the analysis and use advanced vulnerability assessment tools for testing cyber-attacks. This approach will create a more thorough test bed that is similar to real-world systems, allow for power flow analysis and create cyber-attacks that will show vulnerabilities of the system.
Acknowledgement Technical expertise of the test bed has been provided by Iowa State University graduate students Adam Hahn, Aditya Ashok and Siddharth Sridhar. DigSilent expertise has been provided by Iowa State University graduate student Jie Yan.
Problem Statement
General Problem Statement
Our goal is to improve the cyber security of SCADA systems by making our own SCADA test bed, where we can simulate power systems and the communication protocols they use, and attempt cyber-attacks on our systems. Through this process, we can test vulnerabilities of commercial SCADA protection products report their vulnerabilities. We can also demonstrate the effects a SCADA cyber-attack can
SDMAY11-11 5
have on a power system. We will be improving the test bed created by the previous year’s team. We will be expanding the test bed’s number of nodes, adding power flow analysis, and creating more advanced cyber-attacks.
General Solution Approach The three main tasks, as described in our problem statement, are to expand the test bed by having more nodes, add power flow analysis functionality and create and test more advanced cyber-attacks. In order to expand the test bed, we will use virtualization to create more nodes without the need for hardware for each node. This will include virtualization of the relay and RTU. To add power flow analysis to the test bed, we will use software that can connect to the test bed and provide analysis along with providing real world scenarios for the test bed. With regards to the cyber-attacks, we will use vulnerability testing tools to scan for vulnerabilities and then try attacks against the vulnerabilities.
SCADA System with
Poor Security
Improvement Cycle
SCADA System with
Improved Security
System Configuration
and Improvement
Vulnerability Assessment
Attack Scenario
Figure 1: Design Cycle Diagram
SDMAY11-11 6
Operating Environment The operating environment for the test bed is a lab in Coover Hall. The conditions in the lab are normal operating conditions for the test bed equipment.
Intended Users and Uses
Intended Users The primary users of this system will be graduate and undergraduate students in computer engineering or electrical engineering who are researching the cyber security of SCADA systems. Other users of this system might be researchers or companies interested in learning more about the test bed and its functionality.
Intended Uses The primary uses of this system will be the creating and testing of cyber-attacks and researching the
effects that a cyber-attack could have on a SCADA system, especially in regards to power flow. Another
use of this system might be showing people the basics of how a SCADA system works.
Assumptions and Limitations
Assumptions List All test equipment will function correctly
The test bed is similar to a real-world SCADA system
o 15 substations in the test bed will be enough to create real-world scenarios
A pfSense firewall solution will be able to function like a SCALANCE device.
The test bed will demonstrated to those interested in SCADA systems and cyber-security.
Industry might be interested in vulnerabilities found through the test bed.
The test bed will be used in the next years for continuation of cyber-security attacks on a SCADA
system.
Limitations List We have two semesters to complete the project
Only 120V will be used by the relays instead of higher voltages in the real-world such as 330KV.
Only 2 physical relays will be used due to financial limitations
Expected End Product and Other Deliverables At the end of the project period we expect to have a test bed that can be used both for demonstrations and for development of cyber security attacks. This test bed will have over 15 nodes, mostly virtual, with some physical. It will also have the ability to have power flow analysis so it can be used to track the effects a cyber-attack has had on the system. We will also have created cyber-attacks that can be used on the system and demonstrate vulnerabilities.
SDMAY11-11 7
Approach Used
Design objectives
Create a SCADA Testbed that can be used to simulate cyber attacks
o This testbed will allow us to mimic real-world power systems and demonstrate the
effects of a cyber-attack on a SCADA system.
Develop a method to plan, execute, and analyze cyber-attacks on our system
o We want to be methodical in our approach to testing our finished system. It is
important that we have a consistent system that we can use to report our findings.
Functional Requirements
Virtualization
Create a virtualized platform that allows network stack inspection.
o Creating a virtualized platform will be the basis of adding more substations to the
current test bed. Since we are limited on financial resources, we are unable to purchase
more SIPROTEC Relays and SCALANCE devices. We need a virtualized platform that will
allow virtual substations that can connect to the physical test bed. We also need this
platform to have the ability of network stack inspection in order for us to test cyber-
attack scenarios.
Create virtualized images for RTUs, Control Center, firewalls and Relays
o In order to fully virtualize a substation, we will need to create virtual images for each
segment of the substation. Creating a virtualized image for the RTU should be
somewhat basic since it is a software application that runs on Windows. Creating a
virtualized relay will be more difficult since it will require finding a relay simulator that
can communicate with the RTU. We can use an open source firewall solution to simulate
the SCALANCE firewalls.
Virtualized system should be scalable to provide more realistic scenarios.
o We want this system to be scalable to upwards of 30, if not more, substations. To be
able to do this, we will first need to purchase and install a physical virtual host server
with properly allocated physical resources. The substations should be deployed from the
server.
Power System Simulation and Integration
Integrate DIgSILENT PowerFactory with SCADA test bed
o DIgSILENT PowerFactory has the power flow simulation capabilities that we need for our
system. We can set breakers and other components on a PowerFactory schematic to
correspond to data points stored on our SICAM terminals. We will link PowerFactory
and our SICAM RTU’s together via OPC protocol.
Power Simulation should represent real world scenarios
SDMAY11-11 8
o We want to integration between the Power Flow Simulation of PowerFactory and the
test bed to be able to represent real world scenarios. This will make the test bed more
realistic and applicable to the world’s SCADA systems.
Cyber Security Assessment
Produce report detailing security vulnerabilities of the system o The report will detail each vulnerability found during the assessment, what the possible
impact an attack would be if carried out using a particular vulnerability, as well as possible countermeasures to mitigate the effect of each attack.
Shall implement attacks discovered during the vulnerability assessment o We will think of scenarios where an attacker could use a particular vulnerability to
attack the system, try to implement that attack, and attempt to get the attack to work on a consistent basis.
Design Constraints We have a few minor requirements that we have deemed “non-functional”:
Minimal configuration on virtual image deployment
o We want our system to be easy to set up and analyze. We don’t want to have to
configure each of our virtual images individually.
Images should have backups to prevent loss
o We are currently using one external hard drive to accomplish this task, but we are
looking into other solutions.
Attack scenarios can be demonstrated without requiring detailed information on attack
functionality
o The simpler we make our system to operate, the easier it will be to demonstrate it to
the Senior Design Review Board and others who wish to see a demonstration. We will
document how to perform each attack, and if possible, create shell scripts or batch files
to automate the attack.
Assessment shall function as comprehensive documentation on the security state of the system
o This assessment will attempt to be as comprehensive as possible during the information
gathering phase, and will thoroughly document any progress made or failures
encountered. This will help any future project teams build upon it the work
accomplished this year, and hopefully let them avoid repeating any work that has
already been accomplished.
All test equipment should function correctly
Power system should be represented in a manner that is easy to understand
o This will help observers quickly and easily understand the implications of a cyber-
security attack. We are considering using a projector to project our system’s one-line
diagram onto a wall. However, we would prefer to create an easy to understand
display— other than a one-line diagram— to represent our system. This could be a
simple program that we create that reads data points off our OPC server and represents
SDMAY11-11 9
them in an aesthetically pleasing and easily understandable manner. This display would
make our SCADA system very easy to conceptualize, and it will make our system look
more attractive and functional to observers.
Technical approach considerations and results
Virtualization Approach
Software Options for a Virtual Hypervisor
o VmWare Server
Advantages
Can get a free license
Can have multiple virtual machines on 1 computer
Disadvantages
Minimal functionality
It runs on top of an operating system so the resources used by the
operating system will hinder its performance
o VmWare ESX
Advantages
Is the operating system for the computer, minimal resource usage and
overhead.
Can get a free license from the university
Can have multiple virtual machines on 1 computer
Already familiar with this software
Software is easily installed on non-server class hardware
Disadvantages
License only lasts 1 year.
o Citrix XenServer
Advantages
Is the operating system for the computer, minimal resource usage and
overhead.
Can have multiple virtual machines on 1 computer
Disadvantages
No free license available, would need to pay for one.
Not as familiar with this software.
o Microsoft HypverV
Advantages
Can get a free license from the university
Can have multiple virtual machines on 1 computer
Is the operating system for the computer
Disadvantages
Not familiar with this software.
SDMAY11-11 10
Software Selection for a Virtual Hypervisor
We chose to use VmWare ESX as our virtualization hypervisor. A team member was
familiar with the software and has used it before. The university also gives us a 1 year license to
the software so there was no need to spend money on the software. It was also easy to install
on a PC even though it usually recommends server-class hardware be used. This software also
allows for virtual machine templates to be used so it would be easier for use to deploy multiple
substations.
Software Options for a Software Relay Simulator
o Delphin-Informatika IEC 61850 Simulator
Advantages
Was developed with use for SICAM PAS and Siemens Relays
Connected and worked with SICAM PAS
Disadvantages
Only 30 day trial, expensive to purchase
Trial did not include full functionality
Based out of Russia, little amount of support.
o SISCO AX-S4 MMS
Advantages
Free educational license
Provides a network stack for communication
Disadvantages
More complex than the other solutions
o SystemCORP IEC61850 DLL
Advantages
Free
Disadvantages
Poor documentation
Did not connect well to our system.
No Support
Software Selection for a Software Relay Simulator
We chose to use the SISCO AX-S4 MMS as the software for simulating relays. At first we thought
the Delphin-Informatika IEC 61850 Simulator would be our selection. It worked well with our system and
was developed for the same hardware and software that we are using. The draw backs to the Delphin-
Informatika simulator is that the trial only lasted 30 days with basic functionality and that the full license
would be too expensive. We did some more research and found the SISCO simulator. The SISCO AX-S4
MMS provides much functionality as a simulator and SISCO provides a free educational license. Even
though the SISCO product is more complex and will take longer to learn, it was the best option.
SDMAY11-11 11
Power System Simulation and Integration Approach
Software Options
o Siemens Spectrum Power TG DTS (Dispatcher Training Simulation)
Advantages
Software already installed in our lab
Software designed to interact with the our system
Disadvantages
Poor documentation
Hard to set up
Technical support period had expired
o DIgSILENT PowerFactory
Advantages
Has OPC communication capabilities
Easy to use
Extensive documentation
Many people in ECpE department use this software
Disadvantages
Requires advanced license
Software Selection
We chose to use DIgSILENT PowerFactory for our power system simulation. It was
becoming apparent that we required technical support from Siemens if we were going to use
Spectrum Power TG DTS. The manuals were not helpful, and they did not contain the
information we needed. This support costs around $20,000 per year—a price clearly out of our
budget. We found that there was a graduate student here at ISU doing something very similar
to our project. He was using an OPC server to control breakers in DIgSILENT PowerFactory.
Since this was exactly what we wanted to do, and we knew it could be implemented, we
decided to go with that. The use of PowerFactory’s OPC capabilities requires an advanced
license that costs around $2,000. Since this was way less than the Siemens support cost, that
was only going to last a year anyway, we decided it would be better to obtain a license that the
whole department could use.
Cyber Attack/Security Approach
Software Options
o Nessus Security Scanner
Advantages
Remote Vulnerability Scanning
Combined the “Document Running Services” and”Document well-
known software vulnerabilities” phases into one scan
Free License available
Disadvantages
SDMAY11-11 12
Is limited by the plugins that have been created
o Various Open Source Tools
Advantages
Usually free
Disadvantages
Not necessarily well documented or supported
Software Selection
The first piece of software used in performing the vulnerability assessment will be Nessus Security
Scanner from Tenable Security. Nessus remotely scans computers for vulnerabilities, both client-side
and server side, through tests that are specified via the software’s plugin architecture. Nessus generates
a report for each computer which contains a list of any vulnerabilities it discovered during the scan, each
categorized by port number and severity level, as well as reports generated by the test plugin itself.
These reports can be viewed directly on the Nessus Server via a web interface, or exported as an HTML
file.
Figure 2: Sample Nessus Workstation Report
SDMAY11-11 13
Figure 3: Sample Nessus Vulnerability List
It is difficult to predict what software will be used to implement the attacks, as the appropriate software
will vary depending on the type of vulnerability. Most, if not all tools will be free and open source,
though we will not exclude commercial software if it will prove useful. An excellent compilation of
common security tools is the Linux distribution called Backtrack 4, which is available for free from its
website.
Testing approach considerations
Virtualization Testing
How and where will testing be performed?
Testing will be performed in the SCADA lab. We will need to verify the virtual server is
running and communications are working.
Exactly what will be tested?
Communications between virtual RTUs and virtual relays
Communications between virtual RTUs and physical command center
How will testing accuracy be determined?
We will check the RTU operations screen and if it shows that both virtual relay and
command center are connected than it is working correctly
What information will be recorded on the forms that will be used to record test results?
We will record what virtual RTUs and virtual relays are not working and record any
errors associated with them.
SDMAY11-11 14
Who will be doing testing and how will it be verified?
Most likely James Parrott will complete tests. Graduate students will also help in the
testing.
Power System Simulation and Integration Testing
How and where will testing be performed?
Testing will be performed in our SCADA lab. We will need to verify that our SCADA
testbed is interacting with and controlling our power flow software.
Exactly what will be tested?
We will need to test each component on our power flow simulation that is linked to our
OPC server and controlled by our SCADA system. These components will mainly be
relays.
How will testing accuracy be determined?
Our testing will be very objective, since the components that we are testing—virtualized
relays—only exist in two states: on and off. Our operator will be sitting at our control
terminal, and he will toggle the status of a relay. If the change is reflected on our
PowerFactory display, and the power flow solution is adjusted accordingly, we know
that the tested component is functional.
What information will be recorded on the forms that will be used to record test results?
Date/Time, name of component tested, location on OPC server, test failed/successful,
comments
Who will be doing testing and how will it be verified?
Most likely Tony Gedwillo will be performing these tests. Our cooperating grad students
will help to verify these results by attempting to operate the system.
Cyber Security Testing
How and where will testing be performed? o In the lab, on the physical substations.
Exactly what will be tested? o We will test the overall security configuration of the system and attempt to
implement any promising vulnerabilities that are discovered.
How will testing accuracy be determined?
SDMAY11-11 15
o If an attack works properly, then it was accurate to call examine that vulnerability
What information will be recorded on the forms that will be used to record test results? The configuration of each device, as well as whether particular attacks
were effective.
Who will be doing testing and how will it be verified? o David Ryan will be doing this section of testing in cooperation with Adam Hahn.
Recommendations regarding project continuation or modification At this point, we recommend that we continue the project as planned. It appears that we will be able to satisfy our functional requirements in the allotted time. We will be able to virtualize RTU’s and relays, connect our power flow software to the testbed via OPC protocol, and execute cyber-attacks on the system. There is no reason to abandon the project, since there was a large initial investment in the equipment used in the lab and we have the time and ability to complete the project as planned.
Detailed Design
Virtualization:
Overview
This part of the project requires us to install a virtualized hypervisor, install virtual RTUs and virtual relays on the server and have them connect to the current test bed. As stated in the software selections, we will be using VmWare ESX for the virtual hypervisor and SISCO AX-S4 MMS as the relay simulator. Below is a figure the shows what our test bed with virtualized substations will look like.
SDMAY11-11 16
Figure 4: System Diagram
Power Flow Simulation and Integration Relevant software and equipment
o DIgSILENT PowerFactory
This is the software we will use to simulate our power system and solve
its power flow. The substations (busses), generators, loads, and relays
that we want to reflect real world scenarios will be modeled through
this software. These components will be represented on a “one line
diagram” (See Figure 1). The relays modeled in this software will be
controlled by our SCADA system via OPC connectivity. This software will
function as our OPC client. With this software, we can show the effects
of a cyber-attack on a power system.
SDMAY11-11 17
Figure 5: One-Line Diagram from PowerFactory
o Siemens Spectrum Power TG
This software will be used to manually control the statuses of the relays
in our system. Here, we can manipulate our power system. This
software functions as a Human Machine Interface, or an HMI.
SDMAY11-11 18
Figure 6: Using Spectrum Power TG to close a relay
o Siemens SICAM PAS
Our virtualized RTU’s will use SICAM PAS software. This software will
provide the OPC server needed to facilitate communications between
Spectrum Power TG and PowerFactory. After connections are
established between SICAM, PowerFactory, and Spectrum Power TG,
SICAM software will mainly be a background system. During an attack
simulation, users will not directly use SICAM software, and observers
will not be aware of its operation. It simply serves as a communications
point.
SDMAY11-11 19
Figure 7: Conceptualization of our testbed's software communicaiton
Cyber Security Vulnerability Assessment This will be a white-box vulnerability assessment. We have complete access to a fully operational test
bed with no danger of causing any harm if we disrupt normal operations. This provides an excellent
opportunity to research and test any vulnerabilities that might disrupt normal operations in a functional
real-world system.
This assessment will concentrate on the assessing the physical substations because they have a well-
established that will likely change very little in the near future. Any work assessing the physical
substations should carry over into the Virtualization and Power Flow Simulation portions of this project.
The virtualization component will attempt to emulate the physical substations, and the power-flow
simulation should interact the same way with physical or virtual substations.
The testing procedure is as follows:
SDMAY11-11 20
Validate the System
The initial step will be to do a network survey to validate the network, and eliminate any
incorrect assumptions from being made due to incorrect or outdated documentation. A
reference spreadsheet will be created to record all available information about each device. We
will then physically verify that all Ethernet connections are going to the proper place according
to the network map. Last, we will record the host names and IP addresses of all machines in the
lab, as well any software applications that are installed on each machine.
Document Running Services
The next step will be to find out how many ports were exposed to the local network, and what
services were running on each port. This step will be accomplished Nessus Security Scanner.
Nessus will scan through each possible TCP and UDP ports on each computer or hardware
device, detecting whether or not each port responds when queried with traffic. If the service
isn’t directly identifiable to the port scanner, software named Active Ports can be used to
discover which executable opens which port. This information will then be recorded to use as a
reference guide, in case we ever need to readily identify a particular port number or service.
Document Well-Known Software Vulnerabilities
During the port scan, it also runs numerous tests on each port to determine if each port is
susceptible to a particular vulnerability of any severity level.
The client side software scan requires a credentialed scan using Nessus’s SMB logon capabilities.
When Nessus is provided with the local Windows account credentials, the software is able to
check the patch levels of all software on the computer, including Windows itself. Information
about the OS patch level will be added to the reference spreadsheet.
Search for Implementation Vulnerabilities
The final step will be to search for vulnerabilities that are undocumented or specific to our lab
implementation. This includes investigating the Siemens software because Nessus does not have
any tests to evaluate its security level, as well as searching for any weaknesses in
communication or authentication protocols used by any devices or software in the lab.
Attack Implementation
To evaluate the results of the vulnerability assessment, we will attempt to implement any
promising vulnerabilities that are discovered. We will also attempt to make repeating these
attacks as simple as possible by documenting the steps on how to perform the attack, and if
possible, create shell scripts or batch files to run the attack commands.
Produce Report
We will produce a report detailing the existing vulnerabilities of the system, the possible impact
if an attack were carried out using a particular vulnerability, as well as possible countermeasures
to mitigate the effectiveness of a given attack.
SDMAY11-11 21
Project Team Information
Faculty Advisor Information Dr. Manimaran Govindarasu 3227 Coover Ames, IA 50011-3060 Phone: 515-294-9175 Fax: 515-294-3637 Email: [email protected]
Team Information James Parrott Computer Engineering 2132 Sunset Ames, IA 50014 Phone: 515-480-8149 Email: [email protected] David Ryan Computer Engineering 2304 Wallace Rambo Ames, IA 50012 Phone: 563-380-1259 Email: [email protected] Tony Gedwillo Electrical Engineering 6212 Frederiksen Ct Ames, IA 50010 Phone: 402-896-9046 Email: [email protected]
Closing Summary The goal of our SCADA test bed is to mimic real world SCADA systems and to discover and document
vulnerabilities that industrial SCADA systems may have. If industrial SCADA systems are compromised,
money and lives can be lost, especially for large scale SCADA systems like electrical power transmission
systems. We will use virtualized relays and substations (RTU’s) along with control system software and
power flow simulation software to model a SCADA system. Once this system is set up, we can complete
vulnerability assessments, conduct attack scenarios, and document the effects on our power system and
the failures of our security measures. Our hope is that we can provide the power industry, along with
any industry that utilizes SCADA systems, with reports on SCADA system vulnerabilities, so that
preventative measures can be taken.
