Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Data Center Restoration Incident Action Plan (IAP)Action Plan (IAP)
Powerpoint TemplatesPage 1
Powerpoint Templates
Objectives• Understand the roles and responsibilities for the major
players on the Incident Management Team• Understand the components of the Incident Action Plan
(IAP)(IAP)– 202 (Objectives)– 203 (Organization Assignments)– 204 (Assignments)– 205 (Communications Plan)– 206 (Medical Plan)
Additi l D t (WTX 214 U it L )
Powerpoint TemplatesPage 2
– Additional Documents (WTX, 214 Unit Log)• Understand Span of Control
Prerequisite Information
• IS-700 NIMS an Introduction
• IS-100 Introduction to Incident Command
• IS-200 ICS for Single Resources and Initial Action Incidents
• ICS-300 Intermediate ICS for Expanding Incidents (Class room based training)
Powerpoint TemplatesPage 3
Incidents (Class room based training)
3 Types of IAP’s
Powerpoint TemplatesPage 4
Written
Powerpoint TemplatesPage 7
Incident Command System (ICS) and DR
• ICS was developed from the knowledge th t ff ti lthat no one person can effectively manage all of the facets of a complex, rapidly developing situation.
• The IAP ensures that everyone is working toward the same goals set for that
Powerpoint TemplatesPage 8
toward the same goals set for that operational period
NotificationsRecognize andIdentify Problem
Reflex Time
Flow from Event Initiation to Recovery Implementation
Trigger
• 2 am July 3
• Command Center goes dark
• Operators look for a light source to move
• What was that?????
• Get to security office
• Security has no answers
• Got to the data center, it is dark and
• Staff starts calling appropriate persons
• Some are reachable others are
Powerpoint TemplatesPage 9
source to move around
dark and quite
others are not
Split Recovery Processes
DeclarationDamage Assessment Complete
Triage
Flow from Event Initiation to Recovery Implementation
Response
p
• Holiday weekend, key persons away
• Travel time to Data
• Appropriate persons start investigating
• We have a BIG problem – IDATF called in
• IDATF starts assessment
• 2 Hours after arrival recommendation is made to IC
• IC reviews with appropriate persons
• DR plan is executed
• Recovery teams go to DR Site
• Data Center Recovery Teams starts
Powerpoint TemplatesPage 10
Data Center
starts Recovery Incident Action Plan (IAP)
It’s about the TEAM!
• Characteristics– Work well with others under extreme
circumstances
– Capable of working repeated 12 hours shifts
– Persistent goal driven personalities
– Focus on team goals, not personal gain.
Powerpoint TemplatesPage 11
g , p g
– Have a sense of humor
Incident Action Plan• Purpose of the IAP
– Safety of personnel
– Effectiveness of response personnel (Who- What-Effectiveness of response personnel (Who WhatWhen-Where)
• IAP must be accurate enough and complete enough to be a guide for operations
• IAP must constantly be updated, Master during the briefing, updated during the operations
Powerpoint TemplatesPage 12
g, p g pperiod
4 Main Elements of the IAP• What do we want to do? (Objectives)
• Who is responsible for doing it? p g(Organization and Assignments)
• How do we communicate with each other? (Communications Plan)
• What do we do if someone is injured? (M di l Pl )
Powerpoint TemplatesPage 13
(Medical Plan)
NIMS Forms for the Written IAP• NIMS Forms
– 202 (Objectives)203 (Organization Assignments)– 203 (Organization Assignments)
– 204 (Work Assignments)– 205 (Communication Plan)– 206 (Medical Plan)– Additional Documents (WTX, Maps, 214
Powerpoint TemplatesPage 14
Additional Documents (WTX, Maps, 214 Unit Log)
Incident Commander• Responsible to develop SMART objectives (202)
– Specific -Measurable
– Action Oriented -Realistic
Ti S iti– Time Sensitive
• Responsible for choosing team members (203)– Operations Section Chief
– Planning Section Chief
– Logistics Section Chief
Fi Ad i S ti Chi f
Powerpoint TemplatesPage 15
– Finance Admin Section Chief
Operations Section Chief• Develops the strategy to meet the
objectives– Strategy is the bridge between objectives andStrategy is the bridge between objectives and
tactics
• Establishes the tactical control points– How will I break this elephant of a problem
down?– Who will I have manage each control point?
Powerpoint TemplatesPage 16
g p– This breakdown becomes your 204’s
What the 204 accomplishes• Each Group supervisor knows who works for him/her
and who they report to.• Everyone knows what the operational period is.• Everyone in that group knows how they will• Everyone in that group knows how they will
communicate among themselves and how to call for medical help if needed
• Everyone knows where and when they are to report to• Everyone knows what if any special tools they need to
bring with them• Everyone knows what their works assignments for this
Powerpoint TemplatesPage 17
Everyone knows what their works assignments for this operational period are.
Operations Elephant
Powerpoint TemplatesPage 18
Breaking the Elephant
• Created three Branch Directors– Infrastructure Branch Director
• With 2 Groups
– Network & SAN Branch Director• With 4 Groups
– Servers Branch Director
Powerpoint TemplatesPage 19
• With 5 Groups
Breaking the Elephant
• Infrastructure Branch Director Groups– Electrical Group, responsible for all utility and
emergency power systems including ATS, UPS and PDU’s
– Mechanical Group, responsible for all CRAC’s, support area AHU’s, structural i i l di i d fl
Powerpoint TemplatesPage 20
issues including raised floor
Breaking the Elephant• Network and SAN Branch Director Groups
– WAN Group, responsible for all Telco services including AT&T, VzB, and dark fiber, and all internet facing devices
– LAN Group, responsible for all data center routers, switches, physical cabling, rack and stack
– SAN/NAS, responsible for all tiered storage sub-systems
– Voice responsible for all PBX and IVR services
Powerpoint TemplatesPage 21
Voice, responsible for all PBX and IVR services
Breaking the Elephant• Servers Branch Director Groups
– Linux Group– VMS GroupVMS Group– Unix Group– Unix Group– Windows Group– AIX/MPE Group
• Each group responsible for all servers
Powerpoint TemplatesPage 22
• Each group responsible for all servers within their O/S assignment
Communications Plan 205• Built assuming limited cell service
• Needed to be reliable
• Needed to be simple and license free• Needed to be simple and license free
• FRS –Family Radio Service (UHF Band)– Tested communications from core to property lines
with no adverse issues.
• Command Channel (1-C&G Staff)
• Tactical Channels (3 Branches and Groups)
Powerpoint TemplatesPage 23
• Tactical Channels (3 Branches and Groups)
• Medical Channel (Medical Unit Leader or LSC)
Medical Plan 206• We are not working under normal circumstances or
conditions. Items we take for granted need to be documented.
• Working ongoing shifts 7x24 we may have teams from outside the area they are not familiar with our medicaloutside the area, they are not familiar with our medical services
• Where are the medical aid stations, what channel can they be reached on?
• What are the transportation services in the area, ALS or BLS services
• Where are the nearest hospitals? Do they have Trauma centers burn centers and Helipad’s Phone #’s and
Powerpoint TemplatesPage 24
centers, burn centers and Helipad s Phone # s and addresses are provided
Additional Documents• Pre-Marked Satellite Site Map from internet based services.
– Mark sides of the building– Base of operation location– Key Infrastructure components, exterior transformers, switchgear, fire
hydrants, service entrances, etchydrants, service entrances, etc• Weather Report
– No internet service, listen to radio based weather broadcasts– How will this forecast affect the tactics during the operational period?
What is the outlook for the next operational period• 214 Unit Log
– Basic format to keep track of actions, agreements, or arguments during the operational period
– Should be dropped off at the end of each operational period at a pre-designated location and then reviewed for situational awareness
Powerpoint TemplatesPage 25
designated location, and then reviewed for situational awareness, trending, forecasting, status that may not have made its way back to the ICP by another means.
Powerpoint TemplatesPage 26
SPAN OF CONTROL
• Limits the responsibilities and resources being managed by any individual
• Span of Control should be between 3 and 7
• Greater than 7 you are being overloaded
• If you have less than 3 they can probably
Powerpoint TemplatesPage 27
be absorbed by the superior position
It’s a Process
• Our first IAP is a very high level with many assumptions.
• Using the process, the plan is constantly updated aligning us with our objectives
til i id t
Powerpoint TemplatesPage 28
until incident termination