Control Room Management SCADA and Alarms

Alicia Gibson May 2015

Purpose §  Provide background on the reasons for the

Control Room Management Rule. §  Overcoming the hurdles in SCADA and Alarm

Management. –  Performance –  Compliance

Regulatory Intent of CRM Rule - 2009

§  Why does the Control Room Management Rule exist? – To justify PHMSA’s budget? – To keep consultants in business?

Regulatory Intent of CRM Rule - 2009

§  Create an environment to help assure controllers will be successful in maintaining pipeline safety and integrity

§  Verify that procedures, systems and equipment are well thought out, and function as designed

Because Controllers, fatigue, SCADA, alarm systems, changes, and communications issues contributed to these accidents and harm to the public and environment. 1996   1999  

1999   1999   2000  

NTSB Safety Study 2005

Colonial Pipeline – Reedy River §  36” pipeline with corroded section of pipe ruptured

due to overpressure. §  Restarted 2-3 times because of disagreements

between Controller and Supervisor about whether or not it was a pipe rupture.

§  Controller was covering a different console, and was not current in his training on that console.

§  Inadequate communication about the condition of the pipe and pressure restrictions on the portion of pipe that was corroded.

Colonial Pipeline Reedy River Spill – 30,000 barrels (1996)

Colonial Pipeline – Reedy River §  The National Transportation Safety Board determines that

the probable cause of the rupture of the corrosion-weakened pipeline at the Reedy River crossing was the failure of Colonial Pipeline Company (2) to ensure that pipeline controllers were adequately trained to both recognize and respond properly to operational emergencies, abnormal conditions, and pipeline leaks.

Murfreesboro, TN - 1996

Colonial Pipeline – Murfreesboro §  8” pipeline with maintenance being performed

near station and line was shutdown §  Decided to restart line to complete a delivery §  Controller did not open a mainline valve at

station, but restarted the line §  Pressure began rising and Controller did not

see the pressure rise §  Line ruptured, spilled ≈ 2000 barrels

Colonial Pipeline Murfreesboro – 2000 barrels, 1826 PSI (1996)

Display  

Actual  

Colonial Pipeline – Murfreesboro §  The National Transportation Safety Board determines that

the probable cause of this accident was (1) the failure of the pipeline controller to follow company procedures for operating the pipeline and (2) the failure of the pipeline control and monitoring system to inform the controller of unsafe conditions prior to the rupture.

§  Contributing to the severity of the accident was the delay in recognizing that a leak had occurred, which delayed shutting down the pipeline and isolating the rupture.

§ 

Bellingham, WA - 1999

Olympic Pipeline – Bellingham §  3 people died - two 9 year olds playing in a creek with a

fireplace lighter believed to have ignited the gasoline in the creek – died within 24 hours of their burns, one 18 year old fishing in creek succumbed to the fumes and drowned in the creek

§  5,642 barrels of gasoline released §  Relief valve was tightened down after nuisance issues – not

in proper service §  SCADA system work was being conducted on active

displays – Controllers could not see live data

§ 

Olympic Pipeline – Bellingham §  The Safety Board determines that the probable

cause of the June 10, 1999, rupture of the Olympic pipeline in Bellingham, Washington, was (5) Olympic Pipe Line Company’s practice of performing database development work on the supervisory control and data acquisition system while the system was being used to operate the pipeline, which led to the systems becoming non-responsive at a critical time during pipeline operations.

Olympic Pipeline – Bellingham §  The controller operating the pipeline probably would have been able to

initiate actions that would have prevented the pressure increase that ruptured the pipeline if the SCADA system computers had remained responsive to the commands of the Olympic controllers,

§  The degraded SCADA performance experienced by the pipeline controllers on the day of the accident likely resulted from the database development work that was done on the SCADA system.

§  Had the SCADA database revisions that were performed shortly before the accident been performed and thoroughly tested on an off-line system instead of the primary on-line SCADA system, errors resulting from those revisions may have been identified and repaired before they could affect the operation of the pipeline.

§  Olympic did not adequately manage the development, implementation, and protection of its SCADA system.

§ 

NTSB SCADA Safety Study – 108 pages

 Brenham,  TX,  April  7,  1992  –  Seminole  Pipeline  Gramercy,  LA,  May  23,  1996  –  Marathon  Pipeline  Fork  Shoals,  SC,  June  26,  1996  –  Colonial  Pipeline    Murfreesboro,  TN,  November  5,  1996  –  Colonial  Pipeline  Knoxville,  TN,  February  9,  1999  –  Colonial  Pipeline  Bellingham,  WA,  June  10,  1999  –  Olympic  Pipeline  Winchester,  KY,  January  27,  2000  –  Marathon  Pipeline  Greenville,  TX,  March  9,  2000  –  Explorer  Pipeline  Chalk  Point,  MD,  April  7,  2000  –  Piney  Point  Oil  Pipeline    Kingman,  KS,  October  27,  2004  –  Enterprise  Pipeline  

NTSB SCADA Safety Study §  2005 study of hazardous liquids pipeline

accidents from April 1992 to October 2004 •  NTSB study “10 of 13 hazardous liquids accidents

have potential Control Room involvement”

§  “Principal issue…was the delay between a controller’s recognizing a leak and beginning efforts to reduce the effects of the leak.”

§  Identified five areas for potential improvements

NTSB SCADA Safety Study Five areas for potential improvements

1.  Display graphics 2.  Alarm management 3.  Controller training 4.  Controller fatigue 5.  Leak detection systems

Control Room Management Rule §  Must define the roles and responsibilities of controllers… §  And provide controllers with the necessary information,

training, and processes to fulfill these responsibilities. §  …must also implement methods to prevent controller fatigue §  …manage SCADA alarms §  …assure control room considerations are taken into account

when changing pipeline equipment or configurations… §  …review reportable incidents or accidents to determine

whether control room actions contributed to the event.

Roles  and  ResponsibiliWes  General  

Adequate  InformaWon  -­‐  

SCADA  

Adequate  informaWon  -­‐  Shi[  Change  

FaWgue  MiWgaWon  

Alarm  Management   Change  

Management  

OperaWng  Experience  

Training  

Compliance  

CRM Regulation Sections  

Control Room Management §  Was this rule necessary?

–  We believe it was, due to accidents that compromised public safety and caused environmental damage.

§  Can the CRM Rule help improve pipeline operations?

§  Will there be more regulations related to control rooms? –  We think it depends on whether or not industry avoids

major accidents that receive public attention. –  There have been accidents with control room

involvement since the rule was issued.

2010 Spill and Damages – Marshall, MI  

2010 Explosion, Deaths, Injuries and Damage – San Bruno, CA  

2012 Columbia Gas – Sissonville, WV  

Columbia Gas Transmission - 2012 Probable Cause §  The National Transportation Safety Board determines that the

probable cause of the pipeline rupture was (1) external corrosion of the pipe wall due to deteriorated coating and ineffective cathodic protection and (2) the failure to detect the corrosion because the pipeline was not inspected or tested after 1988. Contributing to the poor condition of the corrosion protection systems was the rocky backfill used around the buried pipe. Contributing to the delay in the controller's recognition of the rupture was Columbia Gas Transmission Corporation management's inadequate configuration of the alerts in the supervisory control and data acquisition system. Contributing to the delay in isolating the rupture was the lack of automatic shutoff or remote control valves.

Columbia Gas Transmission - 2012 After Alarm Rationalization

Alarms = 1.9/hour and 45/day Alerts = 83/hour and 1,145/day

Providing Adequate Information – SCADA §  API RP 1165 - Displays §  Point to Point Verification

–  Safety Related Points –  When equipment is added or replaced and when other

changes are made to field equipment or SCADA displays that affect pipeline safety

§  Testing Internal Communications Plan §  Testing Backup SCADA systems

–  Redundant SCADA systems –  Geographically diverse backup control rooms

Alarm Management §  Review Safety Related Alarm Operations

–  Malfunction Reporting and Return to Service –  Priority - # and Identification –  Stale Data

§  Monthly Alarm Review Meeting –  Review/Analysis/Action/Documentation

§  Verify SR Alarm set points and descriptions –  Annually and when equipment is calibrated or changed –  Consistent and Understandable Descriptions –  Controls for Managing Changes to Alarms – Who? How?

§  Annual Review of Alarm Management Plan §  Workload Assessment

Purpose of Module §  Provide background on the reasons for the

Control Room Management Rule. §  Overcoming the hurdles in SCADA and Alarm

Management. –  Performance –  Compliance

§  Email: ali@pipelineperformancegroup.com