Disaster Recovery Disaster Recovery PlanningPlanning

University of UtahUniversity of Utah

Presentation OverviewPresentation Overview

Lessons learned “In the wake of Katrina” by Lessons learned “In the wake of Katrina” by Louisiana State UniversityLouisiana State University

University of Utah disaster planning for IT University of Utah disaster planning for IT servicesservices

What are other colleges implementing?What are other colleges implementing? Are there shared resources in case of a Are there shared resources in case of a

disaster?disaster?

In the wake of KatrinaIn the wake of Katrina

Lessons learned from the Katrina disaster Lessons learned from the Katrina disaster presented by Brian D. Voss, CIO of Louisiana presented by Brian D. Voss, CIO of Louisiana State University presented at ACUTAState University presented at ACUTA

One Campus’ StoryOne Campus’ Story

Copyright Brian D. Voss, 2006Copyright Brian D. Voss, 2006

Sunday August 28thSunday August 28th

SundayAM – clear, bright, warmSundayAM – clear, bright, warm

- Inland Hurricane Wind-Force Warning- Inland Hurricane Wind-Force Warning

Sunday 2:54 PM first bands hit Baton Rouge, LASunday 2:54 PM first bands hit Baton Rouge, LA

Sunday 11:00PM – Sustained tropical force windsSunday 11:00PM – Sustained tropical force winds

Monday 5:57- winds howl – hurricane force gustsMonday 5:57- winds howl – hurricane force gusts

Monday, August 29thMonday, August 29th

“ “ We’ve been hit, but we made it”We’ve been hit, but we made it” Katrina’s Immediate AftermathKatrina’s Immediate Aftermath

- - Lower LA Parishes swamped by storm surgeLower LA Parishes swamped by storm surge

- LSU survived…just a little damage on campus- LSU survived…just a little damage on campus

- Power restored to campus – 6:15 pm- Power restored to campus – 6:15 pm

Tuesday, August 30thTuesday, August 30th

First confirmed reports of a levee failure First confirmed reports of a levee failure

in New Orleansin New Orleans

LSU contacted about expanding routine from a LSU contacted about expanding routine from a

routine special evacuee facility into a broader routine special evacuee facility into a broader purpose, Medical Triage, Special Needs Facilitypurpose, Medical Triage, Special Needs Facility

First IT needs – Phones, phones and more First IT needs – Phones, phones and more phonesphones

Thursday, September 1stThursday, September 1st

LSU establishes a command centerLSU establishes a command center

- Coordinates information for students, and - Coordinates information for students, and evacuees, as well as directing resources to evacuees, as well as directing resources to where they’re neededwhere they’re needed

A plan for displaced studentsA plan for displaced students

Saturday, September 3Saturday, September 3rdrd

More IT help arrivesMore IT help arrives

- Microsoft and IBM send in disaster support - Microsoft and IBM send in disaster support teamsteams

- National Guard convoy delivers 10 boxes of IT - National Guard convoy delivers 10 boxes of IT support donations to LSUsupport donations to LSU

Tuesday, September 6Tuesday, September 6thth -20th -20th

Classes resume !!Classes resume !! PMAC & Field House eventually wind downPMAC & Field House eventually wind down

- Over 6000 patients treated at the facility during Over 6000 patients treated at the facility during the first 7 daysthe first 7 days

- Temporary IT equipment was dismantledTemporary IT equipment was dismantled

Post-Katrina Disaster Recovery Post-Katrina Disaster Recovery ThinkingThinking

Traditional Disaster RecoveryTraditional Disaster Recovery

- What if my data center is lost- What if my data center is lost

Broader Disaster RecoveryBroader Disaster Recovery

- What if my campus is lost- What if my campus is lost

Survivor Disaster RecoverySurvivor Disaster Recovery

- What if I’m fine….. But everyone around me is - What if I’m fine….. But everyone around me is notnot

Lessons Learned at LSULessons Learned at LSU Have a good stock of networking equipment, and Have a good stock of networking equipment, and

mobile and desktop computing in the storeroommobile and desktop computing in the storeroom Have a strong relationship with key vendorsHave a strong relationship with key vendors Great to have terrific, dedicated, service oriented Great to have terrific, dedicated, service oriented

peoplepeople Architectures count- how divisible are the Architectures count- how divisible are the

components? What’s too-tightly integrated?components? What’s too-tightly integrated? Be prepared to be flexible; adapt, improvise, Be prepared to be flexible; adapt, improvise,

overcomeovercome Disaster Recovery and Business Continuity Disaster Recovery and Business Continuity

Planning is not a luxuryPlanning is not a luxury

Lessons LearnedLessons Learned

Do you have a workable DR plan?Do you have a workable DR plan? Do you know where on campus you’ll go?Do you know where on campus you’ll go? Did you take the necessary back-ups and do you Did you take the necessary back-ups and do you

have them ready to re-produce production files?have them ready to re-produce production files? What vendors will you need to tap- and for What vendors will you need to tap- and for

what?what? How will you quickly re-establish network How will you quickly re-establish network

connectivity? Phone service? Web presence? E-connectivity? Phone service? Web presence? E-mail? Mission critical information systems?mail? Mission critical information systems?

Broader Disaster RecoveryBroader Disaster Recovery Are your off-sites conveniently (and perhaps tragically) Are your off-sites conveniently (and perhaps tragically)

close?close? Do you have arrangements to get key services restored Do you have arrangements to get key services restored

at a distance (web, email, financial, student at a distance (web, email, financial, student information?information?

Hot-sites may be too much $$$$-but can you find Hot-sites may be too much $$$$-but can you find suitable raised floor space to rebuildsuitable raised floor space to rebuild

Can you support your administration “ in exile?”Can you support your administration “ in exile?” Is your “life-boat’ plan portable over larger distances?Is your “life-boat’ plan portable over larger distances? Can you grab your key people? Can you care for them?Can you grab your key people? Can you care for them?

University of Utah University of Utah

Water near the Komas Data CenterWater near the Komas Data Center Currently have an offsite data center in Richfield Currently have an offsite data center in Richfield

Utah – Utah –

EOC Emergency Operations CenterEOC Emergency Operations Center

- Small group of individuals from across campus- Small group of individuals from across campus

Plant Operations, Planning, Logistics, Finance and Plant Operations, Planning, Logistics, Finance and Media Media

- Incident Commander (accessing buildings, - Incident Commander (accessing buildings, roads)roads)

IT Disaster Recovery PlansIT Disaster Recovery Plans

Help Desk

IT Plans

SystemsEmailVoice SystemsDesktopNOCC

ConclusionConclusion

What are other Colleges doing?What are other Colleges doing? Are there resources we could share?Are there resources we could share?

- In case of emergencies- In case of emergencies How are other groups handling sensitive data How are other groups handling sensitive data

such as server passwords?such as server passwords?

Disaster Recovery is an ongoing processDisaster Recovery is an ongoing process

- Update on a regular basis- Update on a regular basis