Upload
jeff-roseman
View
1.010
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
IT C ti it fIT Continuity of OperationsOperations
Mi i i th G B t YMinimize the Gaps Between Your Recovery Capabilities and Functional
Requirements of the EnterpriseRequirements of the Enterprise
Presented ByyJeff Roseman
Director, IT Infrastructure ‐ Patterson Medical
March 10, 2009
Semper Paratus: Are You Ready?Semper Paratus: Are You Ready?
• Annual Budget for Disaster Recovery• Annual Budget for Disaster Recovery & Business Continuity?
• Experienced a Disaster?• Experienced a Disaster?• Declared a Disaster in Last Year?• Disaster Recovery Plan?• Updated DR Plan in Last Year? Let Me
S Y• Tested DR Plan in Last Year?• Business Continuity Plan?
See Your Hands
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 2
y
Semper Paratus (Latin: Always Prepared; U.S. Coast Guard motto)
Disaster Recovery vs.Business Continuity
• Disaster Recovery (DR)• Disaster Recovery (DR)– Evolved from Data Center operations– Strictly a “technical” solutionStrictly a technical solution – Over time, it was realized that recovery of the platforms did not mean recovery of the business
• Business Continuity (BC)– Addresses those “non‐technical” functions th t i d t t b ithat are required to restore business
– Not just actions taken during a disaster– An enterprise‐wide project not just IT
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 3
– An enterprise‐wide project, not just IT
2008 AT&T BusinessC i i S dContinuity Study
• One in five businesses does not have a businessOne in five businesses does not have a business continuity plan in place
• For the third year in a row, the survey finds that nearly 30 % of U S businesses don't considernearly 30 % of U.S. businesses don t consider business continuity planning a priority
• Six out of 10 companies have made some type of p ypbusiness change in the past year, but only 28 % updated their plans
• One‐fourth (28 %) have insufficientOne fourth (28 %) have insufficient storage space
• The vast majority (79 %) have special t f i ti ith
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 4
arrangements for communicating with key executives during a natural disaster
Every Business Needs a PlanEvery Business Needs a Plan• A generic DR plan is better than nothing, but it may stress elements thatA generic DR plan is better than nothing, but it may stress elements that
are less important to your business, or worse, leave out critical aspects• Every organization, regardless of size or industry should have a Business
Continuity Plan (BCP).o u y a ( )• Needs vary from business to business and a good availability plan should
be designed for the individual business's needs– Service Delivery / Call Center / eCommerceService Delivery / Call Center / eCommerce– Manufacturing / Distribution– Multi‐Site & International Operations
• A Business Continuity Plan is the least expensive insurance• A Business Continuity Plan is the least expensive insurance any company can have (especially for small companies, as it costs next to nothing to produce)– Treat it as an investment not an expense
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 5
Treat it as an investment not an expense– Many businesses NEVER recover from a serious incident
Taking Your Business Continuityto the Next Level
• It is a huge mistake to
EnterpriseAvailability
gdevelop a business continuity plan and not make it integral with your daily
BusinessContinuity
with your daily business operations
• Availability planning is an investment in the
DisasterRecovery
Days Hours Minutes
continuing operations of the business
• Transform your Business Continuity y
Data‐Centric Business Function‐Centric
Functionality/Cost/Recovery Time Objective — RTOBusiness Continuity Plan into an Enterprise Availability Plan
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 6
Enterprise Availability PlansEnterprise Availability Plans
• The ProcessThe Process– Understand Your Enterprise Requirements – Prioritize and Map Enterprise Requirements – Minimize the Gaps between Requirements and Capabilities
– Test and Modify the Plan to Prevent Future GapsTest and Modify the Plan to Prevent Future Gaps • The Results
– Incident Management Plan – Focused on Crisis Management
– Business Availability Plan – Focused on Work Area Recovery
– Technology Availability Plan – Focused on Technology Recovery
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 7
U d t d YUnderstand Your EnterpriseEnterprise
Requirements
Document Past Downtime EventsDocument Past Downtime Events• A list of known downtime events Common Downtime Eventsand their associated costs will
help you identify common problems and develop solutions that will improve availability 24/7
Common Downtime Events(My Personal Stream of Misfortune)
that will improve availability 24/7– Power Loss– Communication Outage– Hardware FailuresHardware Failures– Scheduled Maintenance
• Your physical location can have a lot to do with it
Hardware FailurePower Outage
– Multi‐tenant Spaces– Construction– Weather Patterns
gWeather / FloodMalicious ActsFire / BuildingSoftware Failure
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 9
Software FailureOther
Identify Systems And Recovery Procedures
• Disaster Recovery Plan (You already have one right?)• Disaster Recovery Plan (You already have one, right?)• How‐To Guides & Instructions• Technology Profile• Technology Profile
– Team Members & Skill Sets– Systems Diagram
See Appendix for Technology
Profiles Examplesy g– Hardware Inventory– DataComm Inventory
Profiles Examples
– Critical Applications– PBX Configurations– Vendors/Partners
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 10
Vendors/Partners– Vital Records
Define Business FunctionsDefine Business Functions
• Scope of Business Operations• Scope of Business Operations– Locations (Single, Multi‐Site, International operations)– Departments / Teams (How is the company organized?)Departments / Teams (How is the company organized?) – Processes / Tasks (What does the department do all day?)– Schedules (Period Close, Peak Seasons, etc.)– Dependencies (Order processing affected by credit dept.)
• Organization charts and process flow diagrams can really help IT understand the business.
• Are there manual work arounds?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 11
Identify Critical Business RequirementsIdentify Critical Business Requirements
• Document internal key personnel and• Document internal key personnel and functions (who is their backup?) Id if h l• Identify who can telecommute
• Document external contacts• Document critical equipment• Identify critical documentsIdentify critical documents• Identify contingency equipment optionsId tif ti l ti
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 12
• Identify your contingency location
Document Key Internal Personnel and Functions
• Consider which job functions are critically necessary,Consider which job functions are critically necessary, every day, not just in an emergency
• Think about who fills those positions when the primary job holder is on vacationprimary job‐holder is on vacation
• These are people who fill positions without which your business absolutely cannot function – make the y ylist as large as necessary, but as small as possible
• Decide what non‐critical employees should do in the event of a disastershould do in the event of a disaster. If there is no place for them to work, will they be in the way of more critical business functions?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 13
business functions?
Identify Who Can TelecommuteIdentify Who Can Telecommute• Some people in your company might be perfectlySome people in your company might be perfectly capable of conducting business from a home office
• Find out who can and who cannot work remotelyh d h l ff• You might consider assuring that your critical staff
(identified in Step 1) can all telecommute if necessaryy
• This is an easy piece that you can build into your daily operations
• Key personnel who cannot telecommute• Key personnel who cannot telecommute will likely need a workstation at your contingency site
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 14
Document Critical External ContactsDocument Critical External Contacts
• Your business partners and vendors can really make or break p yyour recovery– Build a contact list that includes a contact information and a
description services they providel d i li l lik h i– Include in your list people like the insurance company, attorneys,
bankers, IT consultants, electricians...anyone that you might need to call to assist with various operational issues
– Don’t forget utility companies, municipal and community offices, the g y p , p y ,post office and FedEx/UPS.
• Keep a list of key customers who you will want to notify in an emergency
• Create a “Yellow Pages” of external contacts by function and a “White Pages” by name
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 15
Document Critical EquipmentDocument Critical Equipment • Personal computers often contain critical information (You doPersonal computers often contain critical information (You do
have off‐site backups, don’t you?)• Some businesses cannot function even for a few hours
without a Fax machine (i.e. 25% of orders come by fax)• Do you have special printers you absolutely must have? • What about security and encryption keys?• What about security and encryption keys?• Do you have hardware license dongles?• Don’t forget software – that would oftenDon t forget software that would often
be considered critical equipment especially if it is specialized software or if it cannot b l d
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 16
be replaced.
Identify Critical DocumentsIdentify Critical Documents • You need to have everything available that would beYou need to have everything available that would be
necessary to start your business over again – Articles of incorporation and other legal papers– Insurance policies banking information building lease papersInsurance policies, banking information, building lease papers– HR documents, government mandated records, tax returns– Software Licenses, technical documents and source code
• Remember you might be dealing with a total facility loss• Remember, you might be dealing with a total facility loss• You keep copies of your DR Plan off‐site, why wouldn’t you do
the same for your critical business documents?• Store PDFs of critical documents on a
secure, off‐site server that you can access via the Internet in an emergency
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 17
g y
Identify Contingency Equipment Options
• IT Equipmentq p– Where would you rent computers? – Who can provide equipment such as servers on very short notice?
(i.e. CDW has same day service in Chicago)– Are there components with a particularly long lead time? What are
the alternatives?• Telecom
Does your call center require special equipment?– Does your call center require special equipment?– Can your telecom partner provide you with a loaner?– What is the turn‐around time to set‐up a new phone
system?y• Other Equipment
– Can you use a business service outlet like Kinko’s for copies, fax, printing, and other critical functions?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 18
– Where would you rent trucks, air conditioners, generators, etc.?
Identify your Contingency LocationIdentify your Contingency Location
• This is the place you will conduct business while your primary p y y p yoffices are unavailable– It could be a hotel, an adjacent vacant space, or even someone’s
home for a small businessld b h ffi l i– It could be another company office location
– Or a 3rd party site or mobile service like IBM or SunGard– Perhaps telecommuting for everyone is a viable option. D idi WHERE t d d th d f th b i• Deciding WHERE to go depends on the needs of the business– How much space do you need?– What facilities and services do you need?
Will the facility be available to you in a regional disaster?– Will the facility be available to you in a regional disaster?– What solution will get you back up and running fastest?
• Wherever it is, make sure you have all the appropriate contact information and include
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 19
appropriate contact information and include a map in your BCP
P i i i Y E iPrioritize Your Enterprise Requirements and MapRequirements and Map
into Your Plan
Conduct a Business Impact AnalysisConduct a Business Impact Analysis
• Reveal vulnerabilities and potential risks of• Reveal vulnerabilities and potential risks of worst case scenariosM i f fi• Measure impact on safety, finances, marketing, legal compliance, and quality assurance
• Identify the organization’s business unit processes and the estimated recovery time frame for each business unit
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 21
Identify Risks And ExposuresIdentify Risks And Exposures• We confuse the concept of risk—the probability of success or p p y
failure—with the concept of exposure— what is at stake• From a business continuity standpoint, your risks are what is
likely to fail:y– Hardware failure (minimized with redundant hardware such as dual
power supplies, RAID arrays, clustered servers)– Power failure (UPS and/or backup generator)
C iti l d t t t d i fi f f– Critical documents not stored in fire‐proof safe• Your exposure is what is at stake:
– Lost data and informationL f b i l d– Loss of business, sales and revenue
– Government penalties (IRS, SOX, HIPPA)• Understanding the risks and exposures of the business are
fundamental in setting priorities
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 22
fundamental in setting priorities
Priority MetricsPriority Metrics
• Recovery Point Objective – RPO (data)Recovery Point Objective RPO (data)– The acceptable level of data loss exposure following an unplanned
eventThis is the point in time (prior to the disaster) to hich lost data can– This is the point in time (prior to the disaster) to which lost data can be restored, typically the last backup taken offsite
• Recovery Time Objective – RTO (business process)– The maximum acceptable length of time that can
elapse before the lack of critical business functions severely impacts the viability of the business
– This is the total time required to recover critical services – Measured form the time of disaster to resumption of
critical operations (a.k.a – Maximum Allowable Downtime)
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 23
Prioritize Your RequirementsPrioritize Your Requirements• Are there existing Service Level Agreements (SLAs) in place?• Each business unit should rank their business functions based on most
critical to the organization– Financial Impact
O ti l I t– Operational Impact– Reputation Impact– Regulatory Impact
• What are interdependencies between business units?What are interdependencies between business units?• Set Recovery Time Objectives (RTOs) for business functions and the
applications they depend on– < 4 Hrs– < 24 Hrs– < 72 Hrs– < 7 days
7 14 days
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 24
– 7‐14 days– > 15 days
Prioritization ProcessPrioritization Process
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 25
Associate Business Functions With Applications & Data Sets
• Let the business set the recovery requirements, not the y q ,technical capabilities of the organization– RTO for business function drives RTO for systems– These gaps are natural– Gaps will force the technology to improve to meet
the business needs• Mapping is complicated process
Wh b i i d d i ?– What are business process interdependencies? – What are hardware/software dependencies? – Organize applications in tiers based on business priority
(10 departments X 10 task) X 5 applications X 10 locations = a very– (10 departments X 10 task) X 5 applications X 10 locations = a very complex relationship
• You will quickly learn to– Isolate what are the key resources to recover and in what order
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 26
Isolate what are the key resources to recover and in what order– Build recovery strategies around those priorities
Build a Relationship DiagramBuild a Relationship Diagram
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 27
Evaluate ScenariosEvaluate Scenarios• What are the most likely scenarios you will face?
– Power Loss– Telecom Interruption– Hardware Failure
Severe Weather– Severe Weather• What are the most catastrophic scenarios?
– Regional Incidents– Short Term Loss of Facility Availability– Complete Facility Destruction– Global Pandemic
• As your business changes, so will the scenarios you facescenarios you face
• It’s hard to prepare for every conceivable disaster, so start of with the most common outages and work your way up
• These scenarios will be key in putting together your Enterprise Availability
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 28
Plan
Develop Strategies for Minimizing Risk
• Do nothing; assume the risk• Do nothing; assume the risk• Revert to manual processing• Be self recoverable via multiple sites• Contract with a hot‐site/cold‐site vendor
• Contract a mobile recovery facilityContract a mobile recovery facility• Establish a quick‐ship agreement
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 29
There Is No "One Size Fits All" Solution
• It is normal for there to be a mixture of• It is normal for there to be a mixture of solution types within an organizationB ild l i d l ifi h• Build a solution and plan specific to each business function
• Assume business and technology requirements will evolve over time
• Think scalability• Think flexibility
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 30
Think flexibility
Building Your Enterprise Availability Plan
• So you have you a concept of what you will do in a disasterSo you have you a concept of what you will do in a disaster and tons of supporting documentation, NOW WHAT?
• It’s not enough to just throw it all in a big binder and say DONE!DONE!
• Start off with the Incident Management Plan– In a crisis, it the first step to recovery– Most “good practice” standards specify Incident Response
planning now (Sarbanes‐Oxley, ISO, IEEE, ITIL, Payment Card Industry, etc.)Developing your response as an incident is occurring– Developing your response as an incident is occurring probably will create more stress, cost more, take more time and not be as well executed
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 31
Incident Management PlanIncident Management Plan• The Incident Management Plan (IMP) is your Management Playbook
– An enterprise‐wide action plan to help your senior management effectively and efficiently respond to an incident.
– Your plan includes checklists of required activities, an explanation of roles and a definition of your resourcesy
• Incident Management Coordinator is the Quarterback– Management Action Team – Damage Assessment Team
R T– Recovery Team• Incidents usually require a time‐sensitive response –
if staff don’t know what to do, critical information and options may be lost
• Under stress it is good to know who is capable and permitted to decide time‐critical issues
• Have an Incident Operations Hub (the “War Room”) with specific outgoing channels and messages
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 32
with specific outgoing channels and messages
Incident Management OverviewIncident Management Overview
ControlCommand Communicate
Select Recovery
Plan
ImpactAssess-
ment
PlanExecution
Escalate to Mgmt
Incident Occurs
Mobilize Response
Pending Crisis
Post IncidentAnalysis
RecoveryMgmt.
• Establish command and responsibility for managing the incident then mobilize the response
• Determine the scope of the issue, set priorities, appropriateDetermine the scope of the issue, set priorities, appropriate response, and take control the overall recovery process
• Coordinate internal and external communication
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 33
Business Availability PlanBusiness Availability Plan• Business Availability Plan (BAP) is an action plan
focused on maintaining the availability of critical business processes when situations—ranging from minor outages to major disasters—threaten to disrupt them.
• A detailed series of responses, checklists and action steps to deal with situations that might otherwise affect routine work activities
• Each business unit or department should have• Each business unit or department should have their own plan that meets their particular needs and rolls up into the Enterprise Plan
• Individual plan also allow you to spread the work around and make it relevant to the business process ownersrelevant to the business process owners
• Added Bonus : You’re better prepared to meet regulatory, legal and internal audit compliance requirements, with thorough documentation
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 34
Getting Your Business RestartedGetting Your Business Restarted• Business Function Priorities
– What are the functions most critical to the operation (Consistent with your technology recovery priories?)
– What processes can be done manually?• Facilities
– Where can the employees work?– How do they get there?
• Workstations– What office equipment do we need? – What supplies to we need to function?
• Vital Records– What documents do we need to function?– How do we write and deposit checks?
h i i li ?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 35
– Where is our insurance policy?
Technology Availability PlanTechnology Availability Plan• Your existing disaster recovery plan is a good starting point for building a
Technology Availability Plan (TAP)• It’s a defensive measure that prepares your IT management and team
members to respond to—even help prevent—interruptions• All‐inclusive it covers your entire infrastructure as well as• All‐inclusive, it covers your entire infrastructure as well as
telecommunications, systems, applications and data within the data center.
• A detailed series of action steps, activity checklists, personnel role d fi iti id tifi tidefinitions, resource identification
• Technology recovery priorities• Benefits of a comprehensive TAP
– Better preparedness for IT disruptionsBetter preparedness for IT disruptions – More agile, more effective response – Reduced severity and duration of incidents – Greater ability to mitigate risk—and the
associated increased confidence
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 36
associated increased confidence
L th GLessen the Gaps Between CapabilitiesBetween Capabilities and Requirements
For Most Businesses, 100% Availability Is a Myth
• In a perfect world, you would have 100% availability, but whoIn a perfect world, you would have 100% availability, but who can afford complete redundancy?– Smaller business have tighter budgets, but tend to be less complicated– Large corporations have higher requirements and budgetsLarge corporations have higher requirements and budgets– The mid‐market tends to be in the most challenging position
• The most we can hope for is to lessen the gaps between the needs and capabilities of the businessneeds and capabilities of the business
• How do you make it a reality?– Management Buy‐in and Support– Allocation of Resources– Build Availability into Systems– Hard Work and Persistence
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 38
Selling Availability In Your Organization
• Management EducationManagement Education– Downtime impact on the business– Informed managers make better decisions
• Risks and Exposures• Risks and Exposures• Goal: RTO/RPO acceptance
– What managements needs to approve– Communicate in business terms ($$$)
• Cost of Ownership– Initial costs– Ongoing costs
• Return on investmentR bilit & M U ti
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 39
– Recoverability & More Uptime– Customer Service / Satisfaction
Cost Of Downtime AnalysisCost Of Downtime Analysis
• The more complexThe more complex your environment the more resource i i dintensive and expensive it is to keep availableavailable
• High availability is not cheap, but that is nothing compared to a business interruption
Cost of Prevention
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 40
interruption
Example: Downtime Cost to a $$500M Organization
Cost of Outage = $250K/HrLength of Outage w/o Preparation (5 days) = $10MLength of Outage w/ Preparation (1 day) = $2M
SAVINGS = $8M
Cost of Preparation = $75K/year
Odds of Outage 1 in 25 = 4.0%4.0% x $8M = $320K
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 41
Prevention is actually quite cost effective!
Determining ROI Of AvailabilityDetermining ROI Of Availability
• 'Disaster Driven' ROI Solutions• Disaster‐Driven ROI Solutions… – If Your Business Continuity Solution only Addresses UNPLANNED UNPREDICTABLEAddresses UNPLANNED, UNPREDICTABLE DOWNTIME (Less than 5% of Downtime) it will take a disaster to find ROI
• ROI from Everyday SolutionsIf Your Business Continuity solution also– If Your Business Continuity solution also addresses PLANNED, PREDICTABLE DOWNTIME. (95+% of All Downtime), you'll find everyday ROI
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 42
( ), y y ywithout the disaster!
Achieving Management Buy‐InAchieving Management Buy In• Management support of availability solutions requires g pp y q
understanding the business requirements– What are the drivers of the business? Speak the language of business
not just ITh i h f d i ?– What is the cost of downtime?
– What are the other non‐technical effects of business interruption?• Availability is an investment, not an expense
B ild b i i i il bili l i– Build a business case to invest in availability solutions– What is the ROI from implementing availability
solutions?• Strike when the iron is hot there is no• Strike when the iron is hot, there is no
better time to pitch availability than after an outage (even a small one)
• Build consensus form the bottom up and the top down
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 43
Build consensus form the bottom up and the top down
K C t U d tKeep Current: Update Your Plan to PreventYour Plan to Prevent Gaps from Developing
Putting it All TogetherPutting it All Together
• Your AEP is useless if all the information is scatteredYour AEP is useless if all the information is scattered about in different places
• Make it easy to update• Make plenty of copies and give one to each of your key personnel– Make hard‐copy emergency “grab binders”– Keep copies on USB flash drives
• Keep several extra copies off‐site• Keep several extra copies off‐site– Keep copies at home, in your car, and/or in a safety‐deposit box.
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 45
– Upload a copy to a web‐accessible server hosted off‐site
Communicate, Communicate, Communicate
• Share your plan don’t just lock it in a• Share your plan, don t just lock it in a desk drawer!M k i• Make sure everyone in your company is familiar with the Availability Plan
• Hold mandatory training classes for every employee whether they are on the critical list or not
• Keep availability on everyone's radar
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 46
p y y
Test Your PlanTest Your Plan• You’ve put really good ideas down, accumulated all yourYou ve put really good ideas down, accumulated all your
information, identified contingency locations, put your contact lists in place, but can you pull it off?
• One thing you will definitely learn in the test is that youOne thing you will definitely learn in the test is that you haven’t gotten it all just exactly right
• Don’t wait until disaster strikes to figure out what you should do differently next timeshould do differently next time
• If you make any major changes, run it again• Even after you have a solid plan, you should
test it annually• Run desktop simulations: call your team into a
conference room and run through a mock disaster
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 47
g
Plan to Change the PlanPlan to Change the Plan
• “No battle plan survives contact with theNo battle plan survives contact with the enemy.” ‐‐Helmuth von Moltke the Elder
h d l i d• No matter how good your plan is, and no matter how smoothly your test runs, it is likely there will be events outside your planthere will be events outside your plan– The hotel that was to be your DR site is booked up– A key member of the recovery team is on vacationy y– Your backup tape was defective– The one weekend, you leave your laptop at the office the building burns down
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 48
at the office, the building burns down
Review Revise and RedistributeReview, Revise and Redistribute
• Every time something changes update all copies of your EAPEvery time something changes, update all copies of your EAP– New hardware / new software– More importantly…new business processes
• Constant updating can be time consuming, consider using a software tool to manage and update your plans
• Schedule regular reviews of your plan and stick to theSchedule regular reviews of your plan and stick to the schedule
• Never let it get out of date…It is a living document
• An out‐of‐date plan can be worse than useless: it makes you feel safe when you are anything but!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 49
feel safe when you are anything but!
IT Continuity of Operations: Lessons Learned
• Get out of your comfort zone and focus on the business, not y ,just technology
• Embrace availability as a discipline or methodologygy
• Build higher availability into every project• Business needs will change over time• Think flexibility scalability• Think flexibility, scalability• Strive for continuous improvement• Test frequently
Y d ’ l d illi d ll l i b d• You don’t always need a million dollar solution, but you need an annual budget
• No matter how prepared you think you are, the unexpected ill l h M h O ti i t!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 50
will always happen…Murphy was an Optimist!
Questions & AnswersQuestions & Answers“I always tried to turn every disaster into anI always tried to turn every disaster into an opportunity.” ‐‐ John D. Rockefeller
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 51
AppendixAppendix
Where Else Can I Get Information?Where Else Can I Get Information?
• Web Sites• Web Sites– www.drj.com
ti l i– www.contingencyplanning.com– www.globalcontinuity.com– www.recovery.sungard.com– www.disaster‐resource.com– www.businesscontinuitytoday.com
• Professional Organizations
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 53
• Consultants
More of My Favorite DR PagesMore of My Favorite DR Pages
• Downtime Calculator• Downtime Calculator– www.visionsolutions.com/Solutions/Disaster‐Recovery‐toolkit‐downtime‐calc.aspxy p
• Glossary of Terms– www.continuitycentral.com/DRGlossaryofTerms.pwww.continuitycentral.com/DRGlossaryofTerms.pdf
• Business Continuity and Resiliency Self‐y yAssessment Tool– www.ibm.com/services/us/bcrs/self‐assessment
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 54
Document Collection WorksheetsDocument Collection Worksheets
• Applications• Computer
• Vital Records• Employee Contact p
EquipmentOffi E i t
p yInfoE l C ll T• Office Equipment
• Telecom/Voice• Employee Call Trees• 3rd Party Info
• Office Supplies • Alternate Site Space
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 55
ApplicationsApplications– Business FunctionBusiness Function– Recovery Priority– Application RTO– Manual Procedures in Place– Inter‐dependant Applications– VendorVendor– Version– # Licenses– Install Key– Serial NumberMedia Off Site
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 56
– Media Off Site
Computer EquipmentComputer Equipment• FunctionFunction• IP Address• Description• Service Tag / Code• Warranty expires• OS / Service PackOS / Service Pack• Memory• Hard Drive ‐ number & capacity• Specialty cards• Applications supported• Business function
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 57
• Business function
Telecom/VoiceTelecom/Voice• Site Name• Site Name• Circuit Size• Equipment• Circuit ID• Vendor• Contact Number• Contact Number
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 58
Vital RecordsVital Records• Description• Description• Location• Required By• Responsible Partyp y
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 59
Employee Contact & Call TreesEmployee Contact & Call Trees• NameName• Role / Title• Address• Phone
– Office Phone– Cell Phone– Alternate Phone
• E‐mail– Office E‐mailOffice E mail– Personal E‐mail– Alternate E‐mail
• Expertise / Notes
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 60
• Expertise / Notes
3rd Party Info3rd Party Info• Name• Name• Customer #• Telephone• Contact• Comments• Service / Product Provided• Service / Product Provided• Used in this Recovery Activity
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 61
Alternate Site Space
• Workstation TypeWorkstation Type– Hardware/Software– Phone
• Shared Resources– Phone SystemP i– Printers
– Faxes / Copiers• Seats required by departmentSeats required by department
– Match to RTOs (24 hrs, 72 Hrs, etc)– Not everyone needs to be there Day 1
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 62