Improving Water and Wastewater SCADA Cyber Securityisawwsymposium.com/.../uploads/2012/...SCADA-Cyber-Security_slid… · Education & Training Publishing Conferences & Exhibits Improving

Standards

Certification

Education & Training

Publishing

Conferences & Exhibits

Improving Water and

Wastewater SCADA Cyber Security

2012 ISA Water & Wastewater and Automatic Controls SymposiumAugust 7-9, 2012 – Orlando, Florida, USA

Speakers: Bill Phillips and Norman Anderson

2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA 2

Presenter

• Bill Phillips, PE: Bill specializes in delivery of secure

and reliable process control and SCADA network and

communications systems, cyber security vulnerability

assessment, and facility automation and information

system planning and implementation. Bill has over 30

years of process control and SCADA system experience

and has focused on control system network and

communications cyber security for the last decade. Bill

has a BSEE from Clemson University.


Presenter

• Norman Anderson, PE: Norman has over 5 years

experience in the design and commissioning of Process

Control Systems for the Water Sector. Norman has

provided secure and reliable PLC, SCADA, and Network

hardware and software architecture designs and

provided control system automation solutions for a range

of facilities. Norman has an M.S. in EE from Iowa State

University and an M.S. in Physics from the University of

Florida.


Presentation Outline

• Need to secure control systems– Continuing increase in Cyber Attacks

• Notable Cyber Attacks

• Available Guidance and Resources– Standards

– Design Guides

• Assessment/Design/Implementation/Operation– Determining Risk factors and mitigation techniques

• Our Experience and Examples

• Summary


General Increase In Cyber Attacks

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Nu

mb

er

of

Vu

lnera

bilit

ies

Year

CERT Cataloged Vulnerabilities 1995-2007

Vulnerabilities

0

20000

40000

60000

80000

100000

120000

140000

160000

19

88

19

89

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

Incid

en

ts R

ep

ort

ed

Year

CERT Reported Incidents 1988-2003

Incidents

• General Trend of increase in incidents

and vulnerabilities.

• CERT stopped incident monitoring in

2003.

*Source: CERT Statistics

http://www.cert.org/stats/#vul-year


Reported Incidents by Infrastructure

Sector

Water/Wastewater is #4 on the list and has twice the incident rate of most commercial

facilities.

*Source: Summarized by Infrastructure Sector (RISI, 2010)


Industrial Security Incident Attack

Points of Entry

Many attacks are through local business networks and via remote access. These are two common

connections to industrial networks to allow for machines having email and internet access to

connect to SCADA networks and to allow remote vendors to connect to SCADA networks for

maintenance.

*Source: Summarized by Points of Entry (RISI, 2010)


Financial Impacts

Approximately 23% of the industrial security incidents resulted in damages greater than one

million dollars per incident.

*Source: Reported in the U.S. (RISI, 2010)


Media Coverage

• Pump destroyed at water plant Springfield, ILo Believed to be due to cyberattack (not confirmed by DHS).

o Story covered by news media such as the Washington Post, Fox News, CNN, and MSNBC

o Even though unconfirmed, the utility was in the national spotlight for weeks

• Texas SCADA system hacked and screenshots of HMI releasedo Response to DHS downplay of IL incidento Again carried by major news mediao Used a virtual network connection with the internet with simple password to access network


More Infamous Attacks

• Maroochy Shire Sewage Treatment Plant in Queensland, Australia.

o Attack resulted in approximately 212,000

gallons of raw sewage to spill out into local

parks, rivers, and a nearby hotel.

o The attack was perpetrated by a disgruntled

insider and former Contractor, Vitek Boden,

that previously installed the radio-controlled

SCADA equipment for the plant.

o During the attack period, Boden used a laptop

computer and stolen radio on at least 46

occasions to issue unauthorized radio

commands to the SCADA System (Abrams

and Weiss, 2008)


More Infamous Attacks, Continued

• Stuxnet

• High sophisticated WORM to target Siemens PLCs

• Used to destroy centrifuges used for uranium enrichment

• Deployed using USB flash media devices (thumb drives)

o No external connections does not equal safety

• Showed the weaknesses of Industrial Control Systems

• Duqu (Stuxnet Variant)

• Discovered by Symantec and appears to be a variant of Stuxnet

• Not intended to destroy industrial control systems but to steal information from

them

Native Code Code with virus


Common Vulnerabilities

• Denial of Service (DoS):

– Attempt to make computer network unavailable

– Would slow or shutdown the communications SCADA network

– Mitigation techniques include Firewalls, ACLs, Intrusion Prevention Systems

• SQL Injection

– Attacks SQL databases using vulnerabilities in websites

– Can steal database information or destroy data

– Mitigation techniques include effective patch management, Intrusion Prevention Systems

• DCOM

– Most notable are RPC DCOM and Blaster attacks

– Can take control of computer and install programs, view, delete, etc.

– Mitigation includes use of intrusion detection, packet filtering, and network segmentation, and port blocking


Example Control System Attack

Animation Explains Control System Attack By Remote Attacker


Importance of Security

Why Security is Important at a Water or

Wastewater Facility:

• Critical Infrastructure and Public Safety

o Critical resources

o Downtime can affect life safety

• Operational Reliability and Availability

oAttacks can lead to significant downtime

•Financial Impacts

o Loss of revenue for utility and its customers

o Mitigation and legal costs

•Media Attention

o Loss of public confidence

oStaff intimidation


Available Guidance

• AWWA Roadmap to Secure Control Systems in the Water Sector published in 2008

o Goal is in 10 years to have no loss in critical function due to cyber attack

o Develops a roadmap with goals at the 1, 3, and 10 year marks. Currently in year

4 (mid-term) of program

• ANSI/ISA-99.02.01-2009 Security for Industrial Automation

and Control Systems: Establishing an Industrial Automation and

Control Systems Security Program

o Builds upon global standards ISO/IEC 17799 and ISO/IEC

27001 and addresses the difference needed for industrial

security

o Defines procedures for implementing and assessing secure

industrial control systems


Available Guidance, Continued

• NIST SP 800-82

o Final Version Published:

http://csrc.nist.gov/publications/nistpubs/800-82/SP800-

82-final.pdf

o Goal is to provide a guideline for critical infrastructures to

secure their control systems with the idea to maintain

systems online and operating unlike traditional IT systems.

• NERC – Critical Infrastructure Protection (CIP)

o Numbers CIP-002-3 through CIP-009-4 (18 standards)

related to Cyber security implementation plans

o Covers implementation of management controls as well

as operating procedures for personnel


Available Guidance, Continued

• Cisco/Rockwell Automation – Converged Plantwide Ethernet

(CPwE) Design and Implementation Guide

oProvides design and implementation guidelines

for industrial control systems based on the

manufacturing industry

oGoal is to provide less downtime, higher

security, and optimization of Industrial

Ethernet networks

o Guide provides real network architecture

examples, security methods, and

implementation methods


Securing Networks

• Securing networks requires proper planning to ensure successful implementation. There are four basic stages of planning and implementation for network security:

1. Assessment• Determine Risks and Mitigation techniques

• Risk impact versus cost of mitigation

2. Design• Develop appropriate network architecture and segmentation

(NOTE : Taylor to selected HMI suite TCP/UDP port requirements)

• Choose necessary hardware and software

3. Implementation• Qualified and certified installers and designers

4. Operation and Maintenance• Develop operational procedures for staff

• Maintain network, hardware, and software


Assessment

The Critical Starting Point

• First step for proper network security

• Past Assessments were largely based on RAM-W– This method was not very specific or comprehensive

– Limited guidance was available at the time

• US-CERT Cyber Security Evaluation Tool (CSET)– Developed by DHS to assist in protecting key assets with

assistance from NIST

– Available free from the US-CERT website: http://www.us-cert.gov/control_systems/satool.html [training from Control System Security Program (CSSP) also provided]

– Uses 4 major steps and generates a report based on current industry standards

• Assessment is then used to plan and prioritize mitigation solutions


Typical Large Utility Control System

Network


Typical Small Utility Control System

Network


Typical Small Remote Systems

• No matter the size of the network

there are still critical systems to

protect.

• Process control networks are

inherently different than IT

business networks even though

many components are similar.


Wastewater Utility Control System

Design Example

• Includes redundant WAN connections

• Internet connection for WAN extension to remote facilities & mobile remote access

• Compact resilient core network

• Uses VLANs and firewall sub-interfaces to tailor network architecture to SCADA HMI applications suite requirements and to securely support business network access


Network Segmentation – Using

VLANs

• Network organization secures and helps maintain networks.

• Virtual LANs (VLANs) - Useful for SCADA systems because VLANs define broadcast domains that can be widely separated (i.e. not on the same network segment)

• Can reduce costs, by allowing host on different networks to share layer 2 switches.

• Use 802.1q VLAN encapsulation protocol

• Layer 3 device required to route between VLANs, some Layer 2 devices will support VLANs to some extent.

• VLAN Approach:

o VLAN Range: 1-1005 (normal) & 1006-4094 (extended)

o Don’t Use VLAN 1 (Native VLAN)

o Verify VLAN capabilities of network switches & routers

o Use logical approach

o Incorporate VLAN designations into IP Addresses


Network Segmentation – Using

VLANs (Example)

• Example:

• VLAN 10 – Network Management

• VLAN 20 – SCADA DMZ

• VLAN 30 – SCADA

• VLAN 40 – Security (Video)

• VLAN 50 – Remote User (DMZ)

• VLAN 100 – Public Media WAN (Inter Facility VPNs)

• VLAN 110 – Backup Public Media WAN

• Extensions: (For shared media)

• VLAN 60 – Business

• VLAN 70 – Business Remote User (DMZ)


Network IP Addressing

• Approach:• Use 10.0.0.0 private network Class A for primary VLANs

• Use 192.168.0.0 private Class Cs for routed links

• Incorporate facility & VLAN numbers into IP addresses

• Limit broadcast domains to a single facility

• Primary VLAN Example:• 10.VLAN.Facility.Host/X or 10.Facility.VLAN.Host/X

• X = Subnet Mask bit count

• X (between 24 &30) based on anticipated host count

• WAN Example:• 192.168.1.Y/X

• X = Subnet Mask bit count

• (between 24 & 30) based on number of nodes

• Y (between 0 & 252) = Network Number


Example Firewall Configuration

Specification

• Security Levels - Implicit Deny Lower-to-Higher level

• Interfaces• Typically 3-4 for small to medium size firewalls

• Sub-interfaces can extend that number

• Stateful Inspection• Can drop otherwise legitimate packets that are not part of an active

connection

• Holds in memory variables defining the state of each connection

• State variables include things like source and destination addresses, port numbers, packet sequence numbers

• Access Control Lists• Used to apply access control rules at interfaces

• Format: access-list-number {permit | deny} protocol source {source-mask} destination {destination-mask} [eq destination-port]


Example Firewall Configuration

Specification

• Security Levels• Each Interface & Sub-interface

• Inside – 100 (Most trusted)

• Outside – 0 (Least trusted)

• DMZ – 50

• Access Control Lists• Permit DMZ –to-Inside SCADA specific traffic such as web server,

terminal server and historian traffic.

• Permit VPN LAN-to-DMZ authenticated remote user traffic such as web server, terminal server and historian traffic.

• Remote PLC Connections:

• Consider a Remote PLC DMZ to avoid direct connections between Internet connected PLCs and the SCADA network

• Consider dual Ethernet DMZ PLC interfaces (i.e. separate VLANs) to increase separation.


Domain Controller Implementation

• Use group policies to manage role based access

• Separate controllers required for each domain

• Domain Controller and Active Directory Traffic• Uses Remote Procedure Calls (RPC) and Distributed Component

Object Model (DCOM) which introduce numerous vulnerabilities.

• Should not be permitted across firewall boundaries (i.e. don’t extend the corporate domain into the SCADA DMZ)

• Exception – When a Read-only Domain Controller (drastically reducing port requirements) is used with an IPSec VPN tunnel connection to extend the SCADA domain into the SCADA DMZ

• Generally worth the trouble to ease implementation & maintenance of role based access & remote access using RADIUS authentication


Remote VPN Connections

• VPNs can securely extend WANs using public media & provide secure remote access to mobile staff

• Remote Facility Connections Using IPSec Site-to-Site VPNs• Used to interconnect two or more facility LANs

• Encrypts entire IP packet including endpoint private IP addresses

• Provides, confidentiality, data integrity, origin authentication and replay protection

• Mobile Remote User Connections TLS/SSL VPNs• Uses browser interface to connect mobile remote clients to servers

• Operate at the session level to provide secure client/server connections

• Uses certificates to authenticate servers & clients.

• Uses symmetric keys to provide confidentiality and data integrity


VPN Tunnel with Encryption

2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA

Remote Access VPNs

32


Firewalls for Network Security and

Routing


Converged Plantwide Ethernet (CPwE)

Design & Implementation Guide (DIG)

• CPwE DIG – Developed by Cisco Systems & Rockwell Automation

• Provides detailed guidance & includes LAN configuration alternative testing results

• These figures from the DIG are from the LAN and DMZ design chapters

• LAN resilience alternatives shown & performance comparisons.

• L2&3 QoS settings recommendations

• DMZ Example tailored to SCADA


Designing and Implementation

Roadblocks

• Conflicts faced by utilities– Lack of regulatory driver

– Many competing needs

• Losing sleep each time another event makes the news.

• What to do? Utility staffs are a resourceful bunch and they find a way to address their concerns– Some are able to get funding to specifically address cyber security.

– Others have to be more creative.

• Utilities often lack resources to self-perform SCADA security assessments & improvements planning, design and implementation, as mentioned before, there is help.


Designing and Implementation

Roadblocks, Continued

• Our experience– Some utilities, usually bigger ones, have adopted appropriate

standards and established internal policies, procedures and standards that they apply to each project

– More commonly, the utility hasn’t established comprehensive standards and isn’t aware of the vulnerabilities in their existing systems; but would like to make progress as part of each project.

– Sometimes it’s a grass roots or replacement project which means that they are open to a comprehensive solution but do have budgetary constraints.

– Other times it more like what can we shoehorn into this small incremental project.


Example - Incremental Implementation

Well 10 Control Center3621 Redhill Place

Well 10Booster

PLC Box

Well 10ABuilding Control Panel

6Mb/s (CIR)

58/KDFN/103600/TWCS

20M

b/s

(CIR

)

58/K

DFN

/103

604/

TWC

S

10 BaseTX

100 BaseTX

Layer 3 Switch W/ EIGRP100

ASA 55102801/2611 W/ EIGRP100

Layer 2 Switch

TW TelecomMetro Ethernet

(ILAN)Layer-2 (Bridged)

Well8-SCADA-3000-1

WWTP1SCADA3 (HIST)

W-SCADA-WIN911

CityCenter-SCADA-3000-1

1536Kb/s (CIR)58/KDFN/103610/TWCS

Well15-SCADA-3000-1

W-SCADA-TS

WWTP12850 (EMAIL)

1536

Kb/

s (C

IR)

58/K

DFN

/103

674/

TWC

S

1536

Kb/s (C

IR)

58/K

DFN/103

608/TW

CS

1536Kb/s (CIR)

58/KDFN/103677/TWCS

Plant1-SCADA-2801-1

Plant2-SCADA-2801-1 Plant6-SCADA-2801-1

Plant6-SCADA-2960TT-1

Plant2-SCADA-2960TC-1 Fa0/0


Plant 6 SCADA VIEW

WW-SCADA-SRVR1WW-SCADA-SRVR2

Well10-WATER-2960TC-1

W-SCADA-2960TT-1

Esteem

A

P B

ridge

Proxim

a A

P B

ridge

54mbps

Esteem AP Bridge

Exalt A

P

Bridge

Exalt A

P

Bridge

54mbps

Proxim

a A

P B

ridgeP

roxima

AP

Bridge

54mbps

Well10-WATER-2611-1

SCADA-ASA: 5510s in HA

W-SCADA-SERVER1 (Z400)

W-SCADA-CLIENT1 (WATER VIEW)

RX3I

RX3I

Remote VPN User

Remote VPN User

Remote VPN User

IPSec VPN Tunnel

100 BaseFX 62.5 125mu

PLC(Redundant Quantum)

WasteWater-SCADA-SERVER

PLC

MODBUS-IP-CONVERTER

WASTEWATER-SCADA-VIEW

Dell 2700

Well10-WATER-2955-1

OIT

PLC

OIT

PLC

PLC1A

PLC1B

HP JetDirect Lift Station 14.2

MTU PLC

P1Switch

SERVER

PLC

Switch

PLCPLC Cabinet

Tank 15

W_SCADA_CLIENT2

W DMZ

Esteem AP Bridge

OIU

WELL 15Westside Blvd SE

CITY CENTER

1536Kb/s (CIR)58/KDFN/103609/TWCS

WELL 8

WELL 10 (Control Center)

PLANT 2 Industrial Park

PLANT 1

Remote

Connections

Remote

Connections

Remote

Connections

Plant3-SCADA-2801-1


MODBUS-IP-CONVERTER

PLANT 3Address?

1536Kb/s (CIR)58/KDFN/10367?/TWCS

Plant2-WW-L14_2-2955

1536Kb/s (CIR)58/KDFN/103608/TWCS Remote Connection to BPS 12

ATF

BPS

LS13 PLC

LS21 PLC

Mod

b us+

P3 PLC

LS 8

Esteem

A

ccess Point

54mbps

MODBUS-IP-CONVERTERLS8 PLC

Modbus Serial

Esteem

A

ccess Point

Modbus

54mbps

Esteem AP Bridge

Point-to-Point

Connection to Well 12

WW DMZ

W-SCADA-DEV-1

SHARED DMZ

[FUTURE]

SCADA-DMZ-RODC01(DC, NTP, Anti-virus, WSUS)

WW-SCADA-TS

WW-SCADA-WIN911

WW-SCADA-HIST

SCADA-DMZ-RODC02

TW Telecom

Internet

Well10CC-SCADA_DMZ-2960TT-1

WW-SCADA-CLNT

Well10CC-SCADA-3560-1

W-SCADA-2960TT-1

Tx/Fx

100BASE-TX/100BASE-FXMedia Converter

Proxim

a A

P B

ridge

SCADA-DCPRI

SCADA-DCSEC

SHAREDSCADA

MOSCAD MTU

Modbus

WW-SCADA-2960TT-1


Example – Incremental Installation

• Initial installation can be done

using a single Ethernet switch and

no remote connections.

• Remote connections can be added

in the future when they can be

secured correctly.

•Design supports adding disaster

recovery elements as budgets allow

• Initial equipment can be upgraded

in the future through firmware to add

required additional services such as

high availability.


Example - Single Implementation by

Phased and Sequenced Construction


Keys to Successful Implementation

(Abbreviated Version of a Long List)

• Use equipment with a long useful lifetime and low risk of becoming completely obsolete in the short term.

• Have a budget in mind and idea of the risk/reward of network connected systems and equipment.

• Be aware that equipment cost is not an indication of work costs. A $1000 router could cost as much to configure as a $15,000 industrial router.

• Are staff or service contracts in place to maintain and troubleshoot systems? Systems are only as good as the maintenance done.

• Make sure that good system documentation and training will be delivered with the improvements.

• Set up a secure backup configuration storage mechanism & keep a copy of all addressing, configurations, settings, and software.

• Use qualified integrators having the proper certifications where appropriate.

2012 ISA WWAC Symposium Aug 7-9, 2012 – Orlando, Florida, USA

• A strategy for layering protection mechanisms to reduce the impact of a single mechanism failure

• In addition to the technical and operational controls that can be applied to SCADA systems, defense in depth requires long term organizational management and operations commitment to security for:– Developing security policies, procedures and educational

materials that apply directly to SCADA

– Conducting periodic security awareness, incident response and disaster recovery training

– Ongoing maintenance and upgrade of SCADA security throughout its lifecycle

– Restricting physical access to SCADA infrastructure

41

Defense in Depth


User Access

• Simple user interface.

• Do not allow access to the

start menu or other non-

essential programs

• Do not allow access to the

computer

• Require login credentials with

secure passwords and auto logouts

• Use USB security where ports are

available


Summary

• The jury is in, the threat is real and utilities need to act

• Adequate guidance is available to support standards based cyber security improvements

• The DHS CSET tool and INL assessment support team provide a SCADA focused tool for conducting self-assessments

• Without a regulatory driver, funding continues to be a problem.

• Proper planning, implementation, and maintenance is key for a successful system. Systems cannot be installed and forgotten.

• Utilities are finding a way to make meaningful progress with both funding and solutions.

Documents

Improving Water and Wastewater SCADA Cyber Securityisawwsymposium.com/.../uploads/2012/...SCADA-Cyber-Security_slid… · Education & Training Publishing Conferences & Exhibits Improving