Creating an Actionable Disaster Recovery Plan · 2 Disaster Planning Strategy Presentation Outline ¾Plan Justification ¾Disaster Definitions & Facts ¾Costs of a Disaster ¾Benefits

Creating an Actionable Disaster Recovery Plan

2

Disaster Planning StrategyDisaster Planning Strategy

Presentation OutlinePlan Justification

Disaster Definitions & FactsCosts of a DisasterBenefits of Planning

Building an Actionable Disaster Recovery PlanProgram InitiationRisk AssessmentDetailed Risk AssessmentDisaster Recovery PlanMaintenance PlanTest Plan & Results

3


Plan Justification

4


What’s a disaster?

“A disaster is an occurrence that disrupts the functioning of an organization resulting in the loss of data, loss of personnel, loss of business or loss of time” – Hiatt 2000

5


Disaster FactsCommon IT disasters:

Power outages 28%Storm damage 12%Floods 10%Hardware error 8%Bombing 7%Hurricanes 6%Fires 6%Software errors 5%Power surge/spike 5%Earthquake 5%

6


TermsBusiness Continuity Planning

Advance planning and preparations to ensure continuity of critical business functions

Disaster RecoveryAdvance planning and preparations to minimize loss and facilitate recovery of core IT assets

7


Tangible and Intangible CostsPatient care and patient safetyPaying staff who are idleAdded work, related to manual operationsOther hard cash costsLost businessLost customer loyalty – your reputation!

8


Recovery Planning BenefitsReducing legal liabilityMinimizing potential economic lossDecreasing potential exposure to disasterReducing the probability of a disaster occurrenceReducing disruption to normal operationsEnsuring organizational stabilityEnsuring orderly, systematic, and timely recovery

9


Recovery Planning BenefitsMinimizing insurance premiumsReducing reliance on key individualsIncreasing asset protectionEnsuring the safety of personnel and patientsComplying with legal, statutory, and regulatory requirements

10


Why have the stakes risen?eBusiness transitioned many businesses from 8am-5pm to a 24 x 7 x 365 model.Patient care could be compromised without information systems.Operations are running too lean to transition to manual processes and be able to conduct business ‘as usual’.Technology companies are not maintaining inventories as they once did to provide quick disaster shipment capabilities.New exposures: viruses, cyber-crime, terrorism

11


Getting Approval & FundingHistorical data

The National Climactic Data Center (NCDC) is the “Nation’s Scorekeeper” in terms of addressing severe weather events in their historical perspective (www.noaa.gov)

National initiativesHospital Incident Emergency Command System (HIECS)

Regulatory audit complianceHIPAAJCAHO

http://www.noaa.gov/

http://www.noaa.gov/

12


Building an Actionable Disaster Recovery Plan

13


A Practical Approach

Detailed Assessment

Plan Development

Testing & Maintenance

Risk AssessmentInitiation

Phasing:1. Initiation 2. Risk Assessment3. Detailed Assessment4. Plan Development5. Testing & Maintenance

14


Program InitiationInitiation

Detailed Assessment

Plan Development


Risk Assessment

15


Strategic Objectives & ScopeObjective:

Develop overall Strategic Objectives and Scope for DRP Program

Practical Approach:Develop high-level Business Case to support DRP ProgramGather and review existing documentation related to DRP Identify areas of alignment with other Organization InitiativesDefine Program Objectives and Scope

Deliverables:DRP Program Definition

16


Organizational StructureObjective:

Develop DRP Program Organizational Structure

Practical Approach:Identify Sponsorship, Stakeholders and Program ManagerDefine Program Organization, Roles and Responsibilities Dedicate existing Staff and supplement with External Resources

Deliverables:Identification of Sponsor(s), Stakeholders and Program ManagerDefinition of Program Organization, Roles and ResponsibilitiesInitial staffing of Core Team(s)

17


Communication StrategyObjective:

Establish ongoing Communication Strategy

Practical Approach:Define Communication Objectives, Approach and Channels (e.g. Status Reports, Company Publications, etc.)For each Channel, define Audience, Message, Mechanism, Tactics, Measures and Timing Recommendations

Deliverables:DRP Communication Strategy and Timing Recommendations

18


Program Plan & BudgetObjective:

Define High-Level DRP Program Plan and Budget

Practical Approach:Define and obtain consensus on Approach and Plan for the overall DRP ProgramEstimate DRP Program Cost and Resource Requirements

Deliverables:High-level DRP Approach, Plan and Budget Assessment

19


Kick-Off MeetingObjective:

Facilitate Program Kick-Off Meeting

Practical Approach:Host Program Kick-Off Meeting, obtaining stakeholder consensus on Program Scope, Objectives, Communication Strategy, Plan and Budget

Deliverables:Program Kick-Off Meeting Presentation / AgendaKick-Off Meeting

20


Risk AssessmentInitiation

Detailed Assessment


RiskAssessment

Plan Development

21


Process Risk AnalysisObjective:

Perform Business Process Risk Analysis

Practical Approach:Interview Business and IT Subject Matter Experts (SMEs) to define disaster scenarios, create an inventory of the major business processes, define the impact of an interruption and thetolerance for downtime, and prioritize major business processesComplete Risk Assessment for Business Process

Deliverables:High-Level Business Process Current State DefinitionBusiness Process Risk Assessment

22


Business Process Inventory

Application B0HResults to HISLabs

ADT, OrdersLab

Patient Care

Application A0H

Order requisition to ancillary systemOrdersADTOrder Entry

Patient Care

Applications Used

Downtime Tolerance

Impact of Interruption

Primary Outputs

Primary Processing

Primary Inputs

Business Function

Business Line

Process Dependencies

23


Technology InventoryObjective:

Perform Technology Inventory and Risk Assessment

Practical Approach:Interview IT Subject Matter Experts (SMEs) to identify Technology Assets, define interdependencies and prioritize according to time sensitivity and criticalityAudit existing, relevant processes and proceduresComplete Risk Assessment for Technology Assets

Deliverables:High-Level Technology Asset Current State DefinitionTechnology Risk Assessment

24


Technology Inventory

Technology Assets Quantity Location InterdependenciesDowntime Tolerance Criticality

Applications (1) Application 1Supported Desktops (1) Desktop config 1Networking Infrastructure (1) Network device 1PBX / Telephony (1) Telephony device 1

Total Valuation

25


Detailed AssessmentInitiation

Detailed Assessment


RiskAssessment

Plan Development

26


Detailed AssessmentObjective:

Perform Business Process Gap Analysis and identify Remediation Approaches

Practical Approach:Identify opportunities to prevent a disaster and other “quick-hits”Evaluate existing Policies, Workflow, and IT systemsComplete Detailed Current State DefinitionDefine and assess Remediation OptionsDevelop Recommendations and select Remediation SolutionDefine Future State based on implementation of selected SolutionDefine and obtain consensus on the Objectives, Scope, Approach Plan and Budget for Remediation Approach

27


Detailed AssessmentDeliverables:

Detailed Current State DefinitionRemediation Options and Recommended Solution(s)Future State DefinitionRemediation Estimates and Plan

28


Downtime Tolerance Costs

$$$$$$$$

Implementation Implementation Costs

$$$Costs

$$$

$$$$

$$

Downtime Tolerance (hours)Downtime Tolerance (hours)

1010 22 0024243636 44

29


Disaster Recovery PlanningInitiation

Detailed Assessment


RiskAssessment

Plan Development

30


Plan DevelopmentObjective:

Develop DRP plan

Practical Approach:Deploy “quick-hit” solutions Develop high-level recovery strategies and recovery phasesDefine roles and responsibilities including line of commandDefine disaster assessment and declaration definitions and proceduresDevelop emergency/ evacuation procedures that incorporate DRP activitiesDocument organization, staff and system functions and recovery requirements and procedures

31


Plan DevelopmentEstablish recovery locations and document steps to make functional during a disaster Develop business partner and vendor agreementsDevelop communications plan and identify alternative communication toolsCreate contingency plans for missing people, failed proceduresDocument insurance information and proceduresBuild maintenance schedule and procedures

Deliverables:Actionable Disaster Recovery Plan

32


Plan StructureSection 1: Plan Information

Objective: To provide information that will enable the reader or user of this plan to execute it while fully understanding the intentions and parameters with which it was created.

Content: Scope, Approach, Objectives, Team Organization, Pre-Disaster Action Checklist

33


Plan StructureSection 2: Actionable Recovery Steps ***Disaster: Start

Here***

Objective: To provide a step by step checklist of activities that will be performed in the event of a disaster. This section contains the detail for each disaster level, by business line, by recovery option.

Content: Evacuation Checklist, Disaster Declaration Checklist, Recovery Team Activation Checklist, Level 1 – 4 Recovery Steps for all teams and for all recovery options

34


Plan StructureSection 3: Addendums

Objective: To provide one place to access key information and resources required to efficiently and knowledgeably carry out the actionable recovery steps.

Content: Phone list, Insurance Information, Legal Considerations, Key Communication Messages, Facilities Considerations, Security Considerations, Transportation Options

35


Section 1 - Plan InformationGoal:

Enable the user to execute the Plan while fully understanding the intentions and parameters with which it was created

Contents:ScopeApproachObjectivesTeam Organization Plan Activation ProcessDistributionCommunication StrategiesContingency Plans (missing people or failed procedures)

36


Disaster Event Types

Event Types

Event Level 1 Event Level 2


37


Recovery StrategiesEvent Types

Staff Facility TechnologyProcess



38


Recovery Strategies

App 1 App 2 App 1 App 2 App 1 App 21 Execute manual procedures 1 1 1 1 2 Restore from backup 2 2 13 Failover to redundant systems 1 2

Event Level 1 Strategies

Busi

ness

Ar

ea 1

Busi

ness

Ar

ea 2

Busi

ness

Ar

ea 3

Event Level 1

App 1 App 2 App 1 App 2 App 1 App 21 Strategy 1 1 1 1 1 2 Strategy 2 2 2 13 Strategy 3 1 2

Event Level 2 Strategies

Busi

ness

Ar

ea 1

Busi

ness

Ar

ea 2

Busi

ness

Ar

ea 3

Event Level 2

39


Recovery Team StructureStructured using the team approach

Each team has separate section of the Plan within each outage event levelRecovery teams = operational and technical groups responsible for restoring specific functionsEach team only has the authority to carry out the procedures contained in their section of the Plan

The teams are:Command TeamAdministrative Recovery TeamOperational Recovery TeamsTechnical Recovery Teams

40


Recovery Team Structure

Command Team

Operational RecoveryTeam

Technical RecoveryTeam

Business Function 1 Business Function 2 Business Function 3 Phones Applications Infrastructure

Administrative Team

41


Plan Activation Process

Authority to declare a disaster crucial element of plan:

Assigned to restricted number of individualsOnly group authorized to declare a disaster is the Command Team

Outage AlertOutage Alert

Command TeamCommand Team•• Establish command centerEstablish command center•• Determine disaster levelDetermine disaster level•• Supervise recovery stepsSupervise recovery steps

•• Receive initial alertReceive initial alert•• Determine disaster levelDetermine disaster level•• Activate recovery teamsActivate recovery teams

OperationalOperational TechnicalTechnicalAdministrativeAdministrative

•• Receive notificationReceive notification•• Evacuate areaEvacuate area•• Notify team membersNotify team members•• Activate planActivate plan



42


Section 2 - Recovery StepsGoal:

To provide a step by step checklist of activities that will be performed in the event of a disaster. This section contains thedetail for each disaster level, by business line, by recovery option

In the event of a disaster, “start here”

Contents:Evacuation Checklist (OSHA)Recovery LocationsDisaster Declaration ChecklistRecovery Team Activation ChecklistLevel 1 – 4 Recovery Steps for Command, Administrative, Business Lines and Technical Recovery Teams and for all recovery options

43


Command Team Checklist# Start

Day Start Time

End Day

End Time

Activity Team / Owner

Complete Date/Time

Comments

1. 1 E+ 00:00

1 E+ 00:15

Execute emergency response (fire, tornado, etc.)

ALL Refer to your facility emergency action plan

2. 1 E+ 00:15

1 E + 00:30

Determine the disaster level based on the Event Level Definitions below and proceed to Initiate Activation Checklist

Command

3. 1 E + 00:30

1 E+ 00:40

Notify Administration accordingly

Command

4.

1 E+ 00:40

1 E+ 01:00

Notify and activate the Recovery Team Leads what disaster level is being declared:

- Operational Team - Administrative Team - Technical Team

Command

44


Command Team QuestionsGoal:

Remind staff about key action items that don’t necessarily belong in another checklist

Examples:Need Risk Management?Need Safety Team? Questions about safety procedures, personal injury.Need Purchasing Team? Need to purchase supplies, furnitrue, computers, etc.Need Facilities Team? Issues with HVAC, security, parking, restrooms, coffee?Need Communications Team? Issues with reporters, announcements, etc.

45


Recovery LocationsDRP Locations

HospitalData CenterB900 - Basement

Who Goes Here?SysAdminsNetworkOPSTelecommunications

What Happens Here?Server / Network / Systems AssessmentsBackup restorationsFailover ActivitiesSystem Monitoring

Who Goes Here?DRP Command TeamCommunicationAdministrative Support

What Happens Here?Disaster Level DecisionsIssue ManagementActivity DirectionStatus ReportingCommunication

Command CenterConf. Room G & H

MedicalOffice Building

Resource CenterMOB 605b

Who Goes Here?Application SupportInterface TeamDBAs

What Happens Here?Application Assessments / RecoveryInterface Assessments / RecoveryDatabase Assessments / Recovery

Help Desk /Desktop ServicesMOB 215

Who Goes Here?Help DeskDesktop Services

What Happens Here?Help Desk 1st Level SupportDesktop Deployment / Support

46


Section 3 - AddendumsGoal:

To provide one place to access key information and resources required to efficiently and knowledgeably carry out the actionable recovery steps

Contents:Phone List (staff, emergency contact, vendor)Insurance ChecklistTransportation ChecklistLegal ChecklistKey Communication MessagesSecurity Checklist

47


Test Plan & ResultsInitiation

Detailed Assessment


RiskAssessment

Plan Development

48


TestingObjective:

Perform testing

Practical Approach:Perform conference room test (passive testing)Perform full test (active testing)

Deliverables:Passive Test Plan and Test ResultsActive Test Plan and Test Results

49


Plan StructureSection 1: Testing Plan Information

Objective: To provide information that will enable the reader or user of this plan to execute it while fully understanding the intentions and parameters with which it was created.

Content: Scope, Approach, Objectives, Roles and Responsibilities, Testing Environment and Locations, Assumptions, Known Risks and Issues

50


Plan StructureSection 2: Actionable Testing Scenarios and Steps

Objective: To provide step by step conference room testing activities that address all levels of disasters that are represented in the Plan.

Content: Testing Checklists for Level 1 – 4 Disasters, Issue Management Process

51


Plan StructureSection 3: Testing and Maintenance Schedule

Objective: To provide a schedule that will ensure that the Plan is tested and executed in a conference room setting at least two times per year. And, to develop a maintenance schedule that will ensure that the plan is current and relevant.

Content: Testing Activities and Schedule, Participants, Start Dates, End Dates; Maintenance Schedule, Owners, Due Dates

52


Walk-Through TestIntended to orient and educated stakeholders with the organization and content of the PlanIntended to evaluate the Plan for completeness and accuracy, assuring all information is up-to-dateShould include all stakeholders of the BCP and take 1-2 hours to execute

Example: Walk-through Test Script

53


Conference Room TestsObjectives:

Intended to evaluate the detailed checklists of the DRPBy creating scenarios (Level 1, 2, 3) to test different levels of the Plan, all stakeholders will have the opportunity to review individual checklists in addition to evaluating interdependencies between the checklistsShould include all stakeholders of the DRP and take 2-4 hours

Approach:For each level, develop a scenarioFor each scenario, define Type of Test, Participants, Type of Disaster, Day and Time of Disaster Event, Disaster Incident Description, Impact

54


Discussion ItemsTo start the scenario

Who does what at that time?How long does it take?When is it finished?

What were the disaster event discovery procedures?What notifications need to occur?What documentation needs to be prepared?

How should a system outage be handled?What notification should occur?How do you validate the outage?How do you evaluate the impact on related systems?How do you document the process?

55


Discussion ItemsWhat do you do in the meantime?How long do you continue manual processes? What if it is a hardware related problem? The vendor says it will be three days before it can be resolved. What do you do?

How long can they be used?Do they have adequate staff?How will they operate without access to the web?What should be communicated internally and externally?

What decisions need to be made and how quickly?Company personnel need to use their temporary operating procedures

What steps need to be taken?

56


Active TestsIntended to evaluate the execution of the checklists and ensure everyone is comfortable executing their tasks

Should include all stakeholders of the DRP and take 4-8 hours

57


Test ResultsImperative to track test problems in a Test Problem Log

Problem Number, Problem Description, Assigned To, Action Items

Intended to ensure action is taken on problems or issues that arose during the testing so that each iteration brings you closer to a complete plan

58


Maintenance PlanInitiation

Detailed Assessment


RiskAssessment

Plan Development

59


Ongoing MaintenanceObjective:

Insure DRP plans are maintained on an on-going basis

Practical Approach:Maintain DRP command team and recovery team rolesMaintain Vendor List and Supply ListPerform periodic Internal Audits/Reviews Insure change management processes incorporate DRP plan maintenance

Deliverables:Actionable DRP plans

60


Timeline and Activities2 Weeks Prior to Test:

DRP Coordinator sends a message to all Command Team and Recovery Team Leads indicating the time of the testing and requesting Recovery Team Leads make checklist updatesRecovery Team Leads update checklists and distribute to BCP Coordinator

1 Week Prior to Test:DRP Coordinator updates BIA, Recovery Strategies, DRP and Test Plan

61


Timeline and ActivitiesTesting:

Testing occurs over ½ dayDRP Coordinator facilitates all testing activities

3 Weeks After Test:Updates and other action items identified during testing complete

4 Weeks After Test:New DRP compiled and distributed to all Command Team and Recovery Team Leads and Executive Management

62


Summary

63


SummaryDisaster Recovery Planning is essential

Your approach needs to be practical and the plan needs to be executable

Test much and test often

Ensure the plan is maintained

64


Jonathan ThompsonStoneBridge Group

701 Xenia Ave. South, Suite 170Minneapolis, MN 55416

(763) 923-7900(763) 923-7901 fax

www.stonebridgegroup.com

Documents

Creating an Actionable Disaster Recovery Plan · 2 Disaster Planning Strategy Presentation Outline ¾Plan Justification ¾Disaster Definitions & Facts ¾Costs of a Disaster ¾Benefits