This is a disaster declaration! This is not a test

Embed Size (px)

Citation preview

  1. 1. This Is a Disaster Declaration!This Is Not a Test.Richard DolewskiVP of Business Continuity & Business DevelopmentVelocity Technology Solutions, Inc.
  2. 2. VelocityTechnologySolutionsVelocity the ERP in the cloudcompany hosts and managesERP, related applications, and disasterrecovery services within its virtualprivate cloud environment. Aim is tolower operational costs, increasingservice levels and improving softwareperformance while reducing theburden on IT staff and enablingdeployment flexibility. Our team hasexceptional expertise managingenterprise software 24/7. Combinedwith our proprietary technology andsuperfast infrastructure, we increaseapplication availability, security, andcontrol. As a result, our customers arebetter positioned to accelerateenterprise growth and agility, attractand retain customers, and reducecost, turning their ERP platforms into astrategic asset. Velocity isheadquartered in New York City.www.velocity.cc2March 27, 2013Velocity Proprietary and Confidential
  3. 3. Case Study:Disaster DeclarationThe Storyline: A real-world disaster and how events unfolded Company is in the Manufacturing business Corporate data center supports several salesoffices, distribution facilities and manufacturing plantsthroughout the U.S. I will refer to this organization as The Zed Company33 March 27, 2013 Velocity Proprietary and Confidential
  4. 4. What is a Disaster ? ANYTHING !!!! That stops your business from functioning & that cannot becorrected within an acceptable amount of time.4
  5. 5. What is Disaster Recovery? Reaction to a sudden, unplanned, event that enables an organization to continue critical business functions until normal business operations resume. It is not enough to arrange for hardware replacement; planning must address continuation of business operations, or business continuation.5March 27, 2013 Velocity Proprietary and Confidential
  6. 6. Companies Regularly Deal with Natural andMan-Made Disasters Risk is increasing Disasters happenForrester Research
  7. 7. Disasters Have a Major, AdverseImpactRisk is increasing Disasters happenLoss is significant 93% of businesses that suffer a significant loss of data are out of business within 5 years. (Bureau of Labor) 2011, Ponemon Institute Research Report, Sponsored by Emerson Network Power7March 27, 2013 Velocity Proprietary and Confidential
  8. 8. What Constitutes a Disaster in YourOrganization?Disaster Recovery deliverables are based on:RPO - Recovery Point ObjectiveRTO - Recovery Time Objective8 March 27, 2013Velocity Proprietary and Confidential
  9. 9. The Products of a DR PlanWho will execute recovery actionsWhat is needed to continue, resume, recover or restorebusiness functionsWhen business functions and operations must resumeWhere to go to resume corporate, business & operationalfunctionsHow detailed procedures for continuity, resumption, recovery orrestorationCLASSIC: WHO-WHAT-WHERE-WHEN-HOW9March 27, 2013 Velocity Proprietary and Confidential
  10. 10. Disaster Scenario THE ZED COMPANY10 March 27, 2013Velocity Proprietary and Confidential
  11. 11. Profile: The Zed Company Last DR test 8 months ago 100% successful - 17 hours A special save to facilitate DR test audit Have never tested a mid week recovery Recently switched to a new software backup solution No Generator (Not in budget)11March 27, 2013 Velocity Proprietary and Confidential
  12. 12. Profile: The Zed Company Primary Technical Owner of DR conducted last two DR tests in Phoenix Alternate person ran operations back at the office Tapes sent offsite daily RTO 48 hours RPO 24 hours12March 27, 2013Velocity Proprietary and Confidential
  13. 13. The Disaster: The largest power failure in California history
  14. 14. Startup Issues: The Zed Company Person declaring disaster Not authorized and unfamiliar with password activation procedures. Hot-site begins preparation of hot site, but requests authorized individual to call within the next hour14March 27, 2013 Velocity Proprietary and Confidential
  15. 15. Startup Issues: The Zed Company Hot-site equipment is available 4 hours after initial call Zed Company still trying to locate authorized staff (Alternates not available or no longer employed ) Disaster Declaration fee of $7500.00. Whats this? Usage Charge ? Whats this15March 27, 2013Velocity Proprietary and Confidential
  16. 16. Startup Issues: The ZED Company Staff unable to recall tapes en-routeto hot-site Offsite Storage Vendor will notaccept phone requests No phone requests. You must sign on towebsite to recall tapemediaNo App for retrieving your tapes!16 March 27, 2013 Velocity Proprietary and Confidential
  17. 17. Startup Issues: Zed Company Declared a disaster, but no one shows up ! Somewhere en-route from SD Waiting on staff to arrive Unable to locate client. Voice mail. 8 Hours (Dead cell Phone) Client returns call 2 hours later! Power is slowly coming on in parts of the city He has heard his Power may come on shortly 10 Hours17March 27, 2013Velocity Proprietary and Confidential
  18. 18. Startup Issues: The Zed Company Hot-site suggests tapes get shipped to start recoveryjust in case. Client agrees. Client does not know which tapes are required. All Backup Software recovery reports are kept onthe system Suggest sending last full & all of this weeks dailies.Hot-site staff try to recover 12 Hours18 March 27, 2013 Velocity Proprietary and Confidential
  19. 19. Startup Issues: The Zed Company Tape media arrives - a box of 14Backup tapes in NO particular orderNo media report arrived with tapesWhich one contains the catalog? Staff loads all tapes into library, inventories usingbackup software DR Plan missing at hot site - Alternate persongoing back to site to obtain the plan13.5 Hours19 March 27, 2013Velocity Proprietary and Confidential
  20. 20. Startup Issues: The Zed Company Signed into Offsite Tape Vendor website todetermine what backup sets arrived at hot site Found tape catalog as labeled in offsite tape vendorinventory Unable to restore tape catalog invalid catalogerror Manually cataloged each tape 6.5 hours wasted! 20 Hours20 March 27, 2013 Velocity Proprietary and Confidential
  21. 21. Startup Issues: The Zed Company Primary Technical Staff person now en-route to hot-site to assist DR Procedures arrive with Primary person 24 Hours21March 27, 2013Velocity Proprietary and Confidential
  22. 22. Startup Issues: The Zed Company Management are very uptightName calling Every action / or decision is now being questioned Pressure is probably at a peak as Plant locations areasking when ???22 March 27, 2013 Velocity Proprietary and Confidential
  23. 23. Startup Issues: The Zed Company Tapes were cataloged, fortunately the data was found Start recovery of rest of the system32 Hours23 March 27, 2013Velocity Proprietary and Confidential
  24. 24. Startup Issues: The Zed Company Senior management (not IT) summonslead Technical person back as power isnow coming on across the street. Poweris expected anytime now. Declaration is HALTED & tapes arerequested back. Againstrecommendations!34 hours24 March 27, 2013 Velocity Proprietary and Confidential
  25. 25. Back on Again: The Zed Company Client calls back 4 hours later requesting the restore to continue. Power is still an issue. Hot site informs client the system has been scratched and they must start all over again. Client will bring tapes to begin reload. 38 Hours25 March 27, 2013 Velocity Proprietary and Confidential
  26. 26. Back on Again: The Zed Company No staff are available. Everyone exhausted at this point. 40 Hours26 March 27, 2013 Velocity Proprietary and Confidential
  27. 27. Back on Again: The Zed Company Business wants a guarantee the system will be up in 8 hours. Business orders severely affected Issues - Client didnt maintain their backup environment First test since new backup software installed27 March 27, 2013Velocity Proprietary and Confidential
  28. 28. Back on Again: The Zed Company Restore starts immediately, this time with tapes already cataloged and DR documentation in hand System is restored in 17 hours Startup procedures incomplete, but manage to get ERP services up. 62 Hours28 March 27, 2013 Velocity Proprietary and Confidential
  29. 29. System Ready: The Zed CompanyTesters have given the green light.We are back in business.29 March 27, 2013Velocity Proprietary and Confidential
  30. 30. Report Card: The Zed Company Client declares and stops communicating with team Indecisive Backup policies not documented. A mystery to all! New Backup Software recovery never tested Backup Software reporting not sent offsite Tapes shipped. Documentation incomplete30 March 27, 2013 Velocity Proprietary and Confidential
  31. 31. Report Card: The Zed Company Wrong person sent to recover system DR Plan unavailable Senior Management Pressure Staff in a state of flux - Coming & going Halted recovery Restart all over again Lack of staff delegation & training No Generator (Now in budget)31 March 27, 2013Velocity Proprietary and Confidential
  32. 32. Report Card: The ZED Company Does your business have a say? Do the bean counters know what IT can deliver ?32March 27, 2013Velocity Proprietary and Confidential
  33. 33. Report Card: The Zed Company Management shocked that the IT folks lost all orders 24 Hour RPO - Who agreed to this? System was not available for 2 days 48 Hour RTO - Who agreed to this?33 March 27, 2013Velocity Proprietary and Confidential
  34. 34. What about Your Email? The Zed Company Email has become Mission Critical! Email is how you communicate with customers, partners, and employees. In a disaster situation, this may be a key component of your communications strategy. Is Email listed as a critical application for your organization and included in your disaster recovery plan?34March 27, 2013 Velocity Proprietary and Confidential
  35. 35. ModernizeYour Recovery35 March 27, 2013 Velocity Proprietary and Confidential
  36. 36. No one wants to bein a position ofexplaining why yourorganization lost itsdata.36 March 27, 2013 Velocity Proprietary and Confidential
  37. 37. Thank you for attending today: THIS IS A DISASTER DECLARATION. THIS IS NOT A TEST!37 March 27, 2013Velocity Proprietary and Confidential
  38. 38. Presenter: Richard Dolewski VP of Managed Disaster Recovery Services & Business Development [email protected] 206- 436 -3321mcpressonline.com velocity.cc 866-638-277938 March 27, 2013Velocity Proprietary and Confidential