Network Reliability and Interoperability Council

Review of Power Blackout on Telecom

P. J. Aduskevicz, NRSCKathryn Condello, CTIA

Capt. Katharine Burton, NCS/NCC

Network Reliability and Interoperability Council

Power Related Outage DataVoluntary Outage Reporting Trial Data• January – July 2003 Power Related Outage Reports

• 7 of Final Scrubbed Outage Reports• 24% of the Total Final Scrubbed Outage Reports• Examples of Causes Included:

• Back-up Generator Failure, Failed Equipment Power Supplies, and Configuration of Uninterruptible Power Supplies

• Examples of Applicable Best Practices Included: • 6-5-0661 on Routine Exercising of Engines and 6-5-0660 on

Planning for Providing Portable Generators• Outage Reports Associated with the August 14-15, 2003 Power Failure

• 4 Initial Outage Reports received by the NCS/NCCMandatory Outage Reporting Data• Outage Reports Associated with the August 14-15, 2003 Power Failure

• 8 Initial Outage Reports received by the FCC• 2 Final Outage Reports received by the FCC

• Summary Statistics– ~750 NRIC Best Practices

• ~250 Network Reliability• ~250 Physical Security• ~250 Cyber Security + Disaster Recovery + Public

Safety– 98 (13%) with “Power” keyword association

• 72 from NRIC V or earlier• 26 new or edited in NRIC VI

Review of Effectiveness of NRIC Power Best Practices

Quarterly Meeting

August 27, 2003

• History– NRSC Power Studies

• August 2002• 2001• 1996

– California Rolling Blackouts – Finding for Attention

• Less power expertise identified in NRIC VI December 2002 Report as “Area for Attention”

August 2003 Power Outage Event

Quarterly Meeting

August 27, 2003

• Possible areas for Best Practices– Evaluate fuel contractors ability to pump fuel (reports that some pumps

at refueling locations did not have back-up power (work into contracts)– Assess risk for need to enhancing back-up power at cell towers, cable

networks, CLECs, others (implement existing BPs; consider fuel cell technology)

– Consider outreach of sharing best practices to electric power sector– Pre-established plans for coordination between telecommunications

service providers and city, state, and local emergency services– Consumer education on the benefits of having a corded phone

powered from a Central Office

Overall: the traditional wired telecommunications performed well

Review of Effectiveness of NRIC Power Best Practices

Quarterly Meeting

August 27, 2003

The power outage impacted numerous wireless markets. Responding carriers operate nearly 100 switches / 18,500 sites in

blackout area. During first three hours, carriers report demand spikes of 150- 500%

over normal busy hour. All Switches were operational throughout the incident. More than 2/3 of the network was uninterrupted throughout the incident.

Networks were re-configured centrally to provide continued operation while power was replaced and/or augmented.

Backhaul outages (Landline T-1 and DS-3 circuit faults) have been identified as root cause for some service interruptions.

Restoration activities were coordinated through the Telecom-ISAC. Regular Electrical-ISAC updates allowed strategic deployment of

replacement batteries and generators to impacted sites. 77% of Sites were operational at 24 hour mark, and 97% at 36 hours.

Wireless Performance During Outage

% Sites In Service

99%97%

77%69%

0%

20%

40%

60%

80%

100%

0 4 8 12 16 20 24 28 32 36 40

Duration of Outage - Hours

.

8/141530hrs

8/150330hrs

8/151530hrs

8/160730hrs

National Coordinating Center (NCC) & Telecom ISAC

• Established and maintained a conference bridge for wireline and wireless partners for the duration of power outage – Coordinated problem resolution across telecommunications industry

including generator and fueling strategies for New York and Detroit– Coordinated with the Electrical ISAC to identify areas impacted by the

outage and provided updates on restoration efforts

• Manned the ESF 2 position at FEMA HQ• CWIN used as a transport media to coordinate with Canada • SHARES set Operational Level 2 (emergency potential exists)

– Stations active in the affected areas

• TSP not required (circuits were not damaged)• GETS: 1,813 GETS calls into or out of affected areas

– 93% Success rate – GETS designed for 85% call completion success

Wireless Priority Service (WPS) Performance

• Many cell towers had only 2-4 hours of backup battery power

• 48 WPS calls completed

– 12 in the Washington DC area

– 36 in New York/New Jersey

• Operational measurement of WPS call attempts not yet

implemented. – Only call completion information available, based on billing information

• Cell tower access was the major cause of blockage for localized congestion

• WPS IOC capability provides priority to cell tower only offering little improvement during periods of widespread congestion. – Unaware of current limitations, users from FRB New York, New York

City government (NYPD), and GSA New York expressed concern over WPS performance.

Lessons Learned• Power management and restoral practices at the tactical

level under review by carriers – Need to understand root cause of outage– Appropriate mitigating or prevention techniques can be

developed addressing both cause and expected probability of this event occurring again.

– May need modifications to the TESP program to mitigate additional risks

• Need to NS/EP services across ALL wireless Carriers and ALL platforms. – WPS worked, but would have worked better if the NS/EP user

could have had access to the top 5-6 carriers, both GSM and CDMA.

Lessons Learned• Frequent Electrical ISAC teleconference participation

provided excellent situational awareness to the NCC Telecom-ISAC

• Development of TESP program for cellular networks to address priority restoration of power for critical cellular communications facilities is needed

• Additional carriers and WPS FOC capabilities will address widespread outage problem

• Potential training issues on use and capabilities of WPS– Review WPS training to better education users on current and

projected program capabilities