IT Disaster Recovery Plan Development - cbs1.com.my · IT Disaster Recovery Plan Development ... – Plan development DR Infrastructure Develop procedures, ... Some common misconceptions

12-Apr-12Presentation Title | Confidential 1

IT Disaster Recovery Plan Development

Moving Towards Zero Downtime

Presenter : Chor Ming Chong

Note : This presentation is not to be distributed without the express approval of the presenter.

12-Apr-12 Presentation Title | Confidential 2

� This presentation is not to be distributed without the express approval of the presenter.


Agenda

�� What is DRPWhat is DRP

�� DR Plan Development PhasesDR Plan Development Phases–– PrePre--reqreq : BIA/RA/Strategies: BIA/RA/Strategies

–– PolicyPolicy�� Top downTop down

�� Define ScopeDefine Scope

�� Define DisasterDefine Disaster

�� Define SLA Define SLA –– RTO / RPORTO / RPO

�� Define PrioritiesDefine Priorities


Agenda ( continue..)

–– DR Team DR Team OrganisationOrganisation�� Roles and ResponsibilitiesRoles and Responsibilities�� Call TreeCall Tree�� Recovery FlowRecovery Flow�� Command CenterCommand Center

–– Plan developmentPlan development�� DR InfrastructureDR Infrastructure�� Develop procedures, Implementation and Develop procedures, Implementation and

TestingTesting–– Testing & MaintenanceTesting & Maintenance

�� ImportanceImportance�� Life document cycleLife document cycle

–– Closing Closing –– Failure is Not an OptionFailure is Not an Option�� -- Failing to Plan is planning to failFailing to Plan is planning to fail



Business Today

� Operations availability whenever customer needs them, guaranteed

•Relied on 24 hrs a day, 7 days a week, 52 weeks a year

•No single point of failure

•Technology key for success


What Is a DISASTER?❒ A disaster is a unplanned interruption to the busin ess/data center

which renders it inoperable and inaccessible for a prolong period of time and there is a need to activate the contingenc y plan to support the business and meet the customers needs.


… It Won ’t Happen to Us ...

� Some common misconceptions of Disasters..

– It won’t happen to me.– we are safe where we are.

– Chances are remote.

– we have insurance.

– we have vendor support.– not worth the efforts.

– Its too costly.

– cross the bridge when we come to it.


Some Disaster Facts

� US statistics shows for the past 10 years :– 43% of companies affected by major disasters

go out of business– 29% out of those close within 2 years

– 75% of business who lose computer support are no longer to conduct business functions after only 2 weeks

� reported by DRJ,dec97 issue


Some Local Disasters

� Nestle House– Fire, May 1994

� Bata (M) Bhd– Fire, April 1994

� KL Plaza– Structural damage, Jan 1994

� Wisma Stephens– Explosion, June 1993

� Public Bank– Fire, Feb 1992

� KL Golden Triangle– Telecommunications failure,

Aug 1992

� KL Stock Exchange– Network failure, Sep 1993

� Klang Valley– nationwide black out, Sept 30,

1992� Nation wide tremors

– 1993� Port Klang

– Tank explosion, june 1992� Maybank

– System problems, Jan 1988� MAS

– Tower fire, April 1992


Business Loss

LOSS OFBUSINESS

NORMAL BUSINESS CURVE

TIME

EFFORT

NO PLAN

INSURANCE

RECOVERY

PLAN

CONTINUITY

PLAN



Disaster Recovery Plan

❑ Is not BCP…but part of.

❑ A set of predefined plan, procedures

and resources identified so that..

❑ The action to be taken.

❑ The resources to be used..

❑ The procedures to be followed.

❑ … BEFORE, DURING and AFTER

a disaster event).

❑ It defines the scope and recovery time

objectives to ensure that business can

proceed uninterrupted in the event of a

disaster.

❑ It enables a corporate company to stay

in business.


Why Is DRP Required?

REDUCE LIKELIHOOD OF DISASTER

LIMIT THE DAMAGE AND

LOSSES

FULL RESTORATION

A.S.A.P.

ENSURE CRITICAL BUSINESS

AVAILABILITY


Reasons for DRP

➭ Complexity of business and its integration of services makes planning essential

➭ Increased dependencies on IT to deliver products quickly and efficiently

➭ Time is of the essence …the longer you stay away, the more difficult the recovery will become

➭ Prolong outage will result in loss of business competitive edge

➭ Disaster cannot be managed informally


PREEQUISITE To DRP

� Scope of DRP� Level of priority

� Emergency service objective

� Resource requirements� Acquisition strategy

� Interim processing

� Return to business


Basic DR Plan

❒ Identification of individuals responsible for disaster recovery activities.

❒ Organisation of DR teams assigned to specific recovery functionality.

❒ Procedures to recover normal processing functions at the disaster

backup center for business continuity.

❒ Business recovery priority and the expected time frame for recovery.

❒ Identify resources needed for the alternate processing

to meet the service level expectations.

❒ Recovery of the premises and/or assets (such as computers, factory

equipments etc. ).

❒ Or the rebuilding of a new office/factory or computer data center.

❒ IMPORTANT NOTE : A plan is only good on paper unless it is proven to

be working. TEST your plan periodically…it is the only way to know if it

works !!


Recovery Objectives

❒ To give a clearly defined course of action and provide for an orderly and

timely recovery from a major interruption of computing services arising from

a Data Center disaster

❒ To identify personnel, resources and functions necessary for continued operations in the event of a disaster

❒ To identify those systems critical for the business survival and define the

alternative procedures for ongoing support in the event of a prolonged outage

❒ To identify critical resources that would be necessary for temporary

replacement of processing service and resumption of normal computing service

❒ To specify steps necessary to relocate the business to an alternate site if required

❒ To identify customers/clients that must be notified during the outage

❒ To use established plan to resume normal processing, within predefined limits, after a disaster

❒ To document the procedures for safeguarding of critical resources (e.g..

Inventory of hardware, stock, computer equipments, invoices etc..) at a off-site

storage


In Short…DRP

� Describes.– THE ACTION TO BE TAKEN.– THE RESOURCES TO BE USED..– THE PROCEDURES TO BE

FOLLOWED..� BEFORE, DURING and AFTER an

unlikely event that renders inoperative all or part of an organisation’sinformation processing and communication resources..


Types of DRP

� Do nothing !!!� Reciprocal

� Cold – Recovery from scratch

� Warm – Some infrastructure/facilities in place

� Hot – Immediate recovery

LOSS OFBUSINESS

NORMAL BUSINESS CURVE

TIME

EFFORT

NO PLAN Reciprocal

Warm

Hot

Cold



Prevention ..Better Than Cure

❒ DOSH .. Department of Occupational Safety and Hazards

❒ Employee training and awareness❒ Environmental protection

❒ Hazards, health etc❒ Protection of assets❒ Critical documents and ❒ Information technology

❒ Hardware, Software & network and data communications



Pre-Req

❒ Do your Business Continuity Planning

❒ Business Impact Analysis

❒ Define Impact and identify critical core business that will severely impact company’s survival in the event of a disaster

❒ Provide quantitative/qualitative facts to support the need for a BCP

❒ Define your recovery time objective and recovery point Objectives

❒ Risk Analysis

❒ understand what risks pose the gravest threat to your business asset

❒ recommends cost-effective safeguards

❒ Recovery Strategies

❒ None/reciprocal/cold/warm or hot

❒ Build own or subscribe or hybrid


Policy and Key Activities

❒ Top Management commitmment to establishment and maintenance of a comprehensive, viable and practical Disaster Recovery Plan.

❒ States what is required for Corporate survival should the data centers be subject to a disaster.

❒ Define Scope of DR Plan.

❒ Define the categorisation of recovery applications ( BIA ).

❒ Address the vital business functions of company by providing for alternative processing methods to handle those applications that are necessary for Corporate survival.

❒ Define data backup and archival strategies.

❒ Establish roles and responsibilities and the priorities to be given to the team for the recovery process.

❒ Define recovery of damage host and return to Normalcy criterias.


Recovery Scope

❒ Defines the scope and breadth of the DR Plan

❒ What location/premises is the DR Plan

covering

❒ What business applications that it is

supporting

❒ What is the alternative site located

❒ What is the strategy

❒ None/reciprocal/cold/warm/hot

❒ What is the Recovery Time Objective

❒ What is the Recovery Point Objective


RTO

� Recovery Time Objective

– Time taken to recovery the services from the time of disaster (declaration) to the time the services is made available to the users at the recovery site

Disaster

Strikes

Declare

Disaster

Resume

Services at

DR site

Return to

Normalcy


RPO

� Recovery Point Objective

– Recovery of Data to a predefined state in time in the event of a disaster

Normal

Data EntrySystem

Go offline

Resume

Services at

DR site

Recover data

Up to predefined

point in time

Declare

Disaster

Return to

Normalcy


1 2

3 4

5 6 7

9

8 10

1 hour

1 hour

1 hour

2 hours

30 min 30 min 1 hr 30 min 5 hr 30 min 10 min

9 hours

1. Declare disaster2. Inform Recovery Center all all teams

to respond3. Set up Help Desk at Primary and

Secondary Command Center4. Inform all branches and HO Depts.

And update every 2 hours5. Obtain backup tapes from

offsite storage6. Transport critical backup data and recovery

personnel to Recovery center

7. Recover operating system and startupcomputer backup center

8. Recover applications and forward recovery9. Recover network communications10. Setup online systems for critical applications

End of phase 1 recovery

Phase 2 recovery will involve the recovery of theonline systems and applications for non-criticalapplications.


Recovery Facilities

❒ Own or Subscribed

❒ Own

❒ Maintenance❒ Upgrades

❒ Testing

❒ Availability❒ Network

❒ Subscribed

❒ Company background

❒ Facilities –

❒ Accessibility

❒ Reliability

❒ Resiliency

❒ Availability

❒ Security

❒ Environmental

❒ Network Infrastructure

❒ Other support facilities

❒ Work area floor space

❒ Cold site

❒ Storage requirements

❒ Value add support

❒ Subject matter experts


Offsite Storage

❒ It is essential for the success of any recovery plan to have a backup copy of your critical business data and documentation and stored in an offsite facility.

❒ Documentation of offsite data

❒ Schedule of backup

❒ Archival frequency

❒ In the event of a disaster, all the data can be retrieved from the archives at the offiste locations for the recovery of the affected data center.

❒ Important : Offsite storage must be :

❒ Protected

❒ Accessible at all times


Command Center

� Provide centralised and coordinated management and control of all recovery and communications during a disaster recovery situation.

� There can be 2 command centers designated:� Command Center 1 for Recovery Management Team . The EMT,

Admin, Audit and the DRC will be stationed here.� Command Center 2 for operations and technical Team to restore the

computer and network operations at the recovery backup site.� Important.

� Identified Location and made know to all.� Available at all times.� Equipped with necessary items to operate 24 hours round the clock.� Communications : Faxes, dedicated phone lines ( predefined

incoming/outgoing/hotline ), video conferencing, TV and radio access for news updates, internet and intranet access. Etc).



Recovery TeamEmergency

Management Team

Disaster RecoveryCoordinator

Audit

Site RestorationTeam

AdministrationTeam

OperationsRecovery Team

User LiaisonTeam

Operations Network Systems Security Applications Database Helpdesk

* The above operations recovery teams will cover the technical recovery processes of the respective systems and applications

ProductionProd.Control

Storage MgmtShift

DISASTER RECOVERY MANAGEMENT TEAM

DisasterRecovery

Management


DR Teams - Overview

❒ Responsible to manage and execute all activities defined in the DR Plan.

❒ An identification of those individuals who would be responsible for disaster recovery activities.

❒ An outlet of duties for the management and recovery teams in the Disaster.

❒ Structured processes involving various areas of recovery.

❒ Achieve timely and orderly manner of recovery.


DR Management Team Structure

❒ Emergency Management Team :❒ Head : CIO or Head of the IT Division/Department.

❒ Reports to the company Chairman and Steering Committee.

❒ Responsible for high-level business recovery operations.❒ Main tasks:

❒ Review impact of disaster on the business.

❒ Declaration of Disaster.

❒ Issue directive to activate DR organisation.❒ Provide a channel for key decision(s) during the recovery operations.

❒ Consists of members from the senior Management, selected corporate Co-ordinators and end-users management.



❒ Consists of the Management Teams responsible for th e various areas defined in the Disaster Recovery Plan.

❒ Head : Responsible for providing overall direction for EDP recovery operations.❒ Main task :

❒ Damage assessment of the computer facilities and environment.❒ Activation of the various DR Teams.❒ Set up command centers.❒ liaise with and provide advice to senior management on recovery progress.❒ Co-ordinate actions of the various recovery teams.

❒ Consists of the following members:❒ DR Coordinator.❒ DR Manager.❒ Administration Support Team Manager.❒ Operations Recovery Team Manager.❒ Application Support Team Manager.❒ Site Restoration Team Manager.



❑ Administrative Support Team.❑ Responsible to Serve as expeditors and suppliers of resources to other

disaster recovery team members.❑ Main tasks :

❑ All DR logistics issues including transportation and lodging.❑ Financial matters (ie cash advances),❑ Security.❑ Public relations.❑ Insurance and legal assistance.

❑ Members :❑ Representative from Personnel.❑ Representative from Security.❑ Representative from Public Relations.❑ Representative from Legal.❑ Representative from Head Office Administration.



❑ Application Team Support : ❑ Responsible for coordinating and providing support between all user

locations and the computer backup site.❑ Main tasks :

❑ Advise branch and HO Depts. On use of interim manual procedures.

❑ Update users on recovery progress.

❑ Assists users to resume normal operations when system is available.❑ Members.

❑ Representative Systems and Methods.

❑ Representative Branch and Operations.❑ Branch Support(Help Desk).


Site Restoration Team

❑ Site Restoration Team : Responsible for all damage assessment and

ascertaining the extent of damage to operations affecting all computing

and data communication facilities. Also in charge of rebuilding new

computer center for business resumption plan inclusive of acquisition

strategy.

❑ Main tasks.

❑ Damage assessment and impact analysis on extent of damage.

❑ Inventory and acquisitions of new computer hardware and network data

communication for replacement or repairs.

❑ Restoration of the damage computer center or rebuilding of a new

computer center.

❑ Members:

❑ Buidling Management/Property.

❑ Security.

❑ Operations.

❑ Network Support.

❑ Head Office Admin.

❑ Vendors.


Operations Recovery Team

❑ Operations Recovery Team❑ Responsible for the technical recovery of the computer and data communications facilities❑ Main task:

❑ Restoration of critical online systems in backup recovery center❑ Restoring data communications ❑ Resuming normal processing functions at backup recovery center

❑ Members❑ Systems❑ Operations❑ Security❑ Applications❑ Network❑ Database Administrator❑ Vendors


❑ Immediate Response Steps

❑ Notification

❑ Alert Team Leaders, members &

backup site

❑ Conduct initial debriefing

❑ Perform damage assessment

❑ To declare or not to declare

❑ Setup command centers

❑ Assemble all team members

❑ Execute DR Plan


❑ Setup command center

❑ Primary Command Center :

❑ Emergency Management Team/Site Restoration Team

/Audit Team/DR Coordinator/DR Manager

❑ Secondary Command Center:

❑ DR Technical /Operations Teams

❑ Recovery on at backup site :

❑ Establish recovery center online systems & network

communications

❑ Recovery at damage data center :

❑ Perform detailed damage assessment process including

salvaging process..etc


� Resume normal online

processing at recovery center

� Support online users

� Batch and report distribution

� Staffing

� Logistics & supplies

� Damage Assessment

– Repair or rebuild new

� Salvaging data and equipment

� Acquisition of hardware,

software etc.

� Reestablishing network

communications


❑ Confirm new host data center ready

❑ Environmental resources

❑ Hardware/software equipment

❑ Network communications

❑ Logistics and supplies

❑ Migration back to new host

❑ Shutdown online system

❑ Recovery online system at new host

❑ Resume normal processing

❑ Recovery cycle complete



PROCEDURES CREED

❒ Procedures for every phase of recovery

from prevention, to recovery and interim

processing and finally resumption of

normal business

❒ Procedures are only good and valid if

properly maintained and ownership

properly identified

❒ Procedures must be made available at

recovery center

❒ Procedures must be tested periodically


❑ The plan cannot be executed without people !!

❑ Identification and training must be provided

❑ Key staff contact must be maintained

❑ Know your people and their roles

❑ People are the key to the success of the Plan

No people no work!!


Measure of Success

Successful testing of a recovery capability

is accomplished by developing and using

a test plan that exercises the recovery

procedures and documentation

….and the participation of the staff

For total success. the two cannot

function without the other


DRP Golden Rules

❑ The Plan must be tested regularly to ensure that it is:

❑ workable

❑ up to date

❑ covers all critical areas as defined in the SLA

❑ Also it is a BNM directive for all financial institutions to test the DR plan at least twice a year

❑ The DR teams must be trained according to the procedures set

❑ Procedures must be maintained up to date and kept offsite

❑ Critical Data must be kept offsite and

readily retrievable in times of needs.

❑ The plans must be reviewed

periodically to ensure that it is up-to-

date and the scope updated as when the

business grows

❑ The backup recovery resource capacity

should be reviewed periodically

❑ Remember.. Your DR Plan is only as

good as the next test!!


12-Apr-12Presentation Title | Confidential 52

Thank You

Documents

IT Disaster Recovery Plan Development - cbs1.com.my · IT Disaster Recovery Plan Development ... – Plan development DR Infrastructure Develop procedures, ... Some common misconceptions