IW308: Windows SharePoint Services and SharePoint Portal Server: Data Protection, Disaster Recovery, Site Archival and Beyond

Joel OlesonJoel OlesonTechnologistTechnologistIT Microsoft CorporationIT Microsoft Corporation

Overview

Microsoft IT Case StudyMicrosoft IT Case StudyMS SharePoint EnvironmentMS SharePoint Environment

IT Pain PointsIT Pain Points

Data Protection / BackupsData Protection / Backups

Disaster RecoveryDisaster Recovery

Data Recovery / RestoreData Recovery / RestoreDemo TextCopyDemo TextCopy

IT ImprovementsIT Improvements

Chofu

Dublin

Singapore

SharePoint Deployment

Redmond

Stats:Stats:10 WSS Farms, 6 SPS Farms10 WSS Farms, 6 SPS Farms7 SQL Clusters7 SQL Clusters2.7 TB of database content2.7 TB of database content~320 SPS 2003 Portals~320 SPS 2003 Portals50,000+ Site Collections50,000+ Site Collections89000 User Profiles89000 User Profiles

Americas; Europe, Middle East, Africa (EMEA); Americas; Europe, Middle East, Africa (EMEA); South Pacific; Far EastSouth Pacific; Far East

4 Regional Intranet Deployments4 Regional Intranet Deployments3 Regional Extranet Deployments3 Regional Extranet Deployments

Americas Intranet Hardware

MSWeb Shar ed Ser vices

2 WFE

2 Sear ch

2 Index

2 WFE

Por tal 100

2 WFE

Por tal 200

2 WFE

Upgr aded WSS Far m

SQL (A/P)

Team SQL Cluster

SharePoint Portal Server 2003 SharePoint Services

2 WFE

Team Far m

2 WFE

Host Header

SQL (A/P)

Por tal SQL Cluster

Regional SharePoint

SQL (A/P)

Consolidated SQL Cluster

Intranet Deployment

SQL 2000

Extranet Deployment

Shar ed Ser vices and Por tal Far m

2 WFE

2 Sear ch

2 Index

2 WFE

Cor p Host ing

2 WFE

Host Header

2 WFE

Extr anet WSS

IT SharePoint Storage4 TB SAN (DB)4 TB SAN (DB)

2 TB per SQL Cluster 2 TB per SQL Cluster (DB) (DB)

Drive RecoveryDrive Recovery

Site Backups Site Backups

OtherOtherTransaction Logs, Quorum Drive &System databases (13%)

Site Dump (29%)

DB Dump/backup (29%)

Databases (29%)

SPReport Results (June 04)Report Tool Available from Microsoft.com

Site Collections: 62000 Sites: 15000Site Collections: 62000 Sites: 15000Total Size: 2.7TB Total Size: 2.7TB Average Team Site Size: 56MB Average Team Site Size: 56MB

Large Sites (250MB+): 1710 (5-250MB): 23875 Large Sites (250MB+): 1710 (5-250MB): 23875 (5MB-): 56049 (5MB-): 56049

Average Per Site StatsAverage Per Site StatsUsed Lists: 4 Used Lists: 4 Alerts: 1 Alerts: 1 Users: 29 Users: 29

Readers: 12 Readers: 12 Contributors: 15 Contributors: 15 Web Designers: 3 Web Designers: 3 Admin: 4 Admin: 4 Customized Home Page: 15% (30% Team)Customized Home Page: 15% (30% Team)

IT Pain Points

TapeTapeTape Backups long runningTape Backups long running300-500MB/min300-500MB/minHigh NIC utilization by Tape backupHigh NIC utilization by Tape backup

DatabaseDatabaseDatabase backups long running Database backups long running

1TB 10hr (100 GB/hr)1TB 10hr (100 GB/hr)

SPSBackup backup duplication of databasesSPSBackup backup duplication of databases

Site CollectionSite CollectionSTSADM doesn’t scale well over the network STSADM doesn’t scale well over the network

~300GB in 24/hr (12MB/min)~300GB in 24/hr (12MB/min)

STSADM full 2-3 days (1TB)STSADM full 2-3 days (1TB)

Restore management painsRestore management pains

Data Protection

What to backupWhat to backup

Database BackupsDatabase Backups

ToolsToolsSPS ToolsSPS Tools

WSS ToolsWSS Tools

Spbackup Reskit demoSpbackup Reskit demo

Third Party Backup VendorsThird Party Backup Vendors

Scripts/Dev IdeasScripts/Dev Ideas

Disk to TapeDisk to Tape

SharePoint Storage Objects

Database

Site Collections

Document Libraries and Lists

Files/Documents and List Items

What to Backup

DatabasesDatabasesWSS/SPSWSS/SPS

CONFIG DBCONFIG DB

STS_W3SVC#_#STS_W3SVC#_#SPSSPS

PROF DBPROF DB

SERVDBSERVDB

SITEDBSITEDB

*External to Db*External to DbSystem StateSystem State

SPS: Indexes & SPS: Indexes & WebStore Doc LibWebStore Doc Lib

IIS Meta baseIIS Meta base

Web Part PackagesWeb Part Packages

Home DirectoriesHome Directories

Database Backups

Database to dump on diskDatabase to dump on disk

Databases TypesSharePoint DatabasesSharePoint Databases

CONFIGDB – CONFIGDB – Config data of Databases, Config data of Databases, Virtual Servers, and Site Collections Virtual Servers, and Site Collections

PROFDB PROFDB -- User profiles databaseUser profiles database

SERVDB SERVDB – Alerts & Search scopes– Alerts & Search scopes

SITEDB SITEDB - Portal's content, areas, site - Portal's content, areas, site collectionscollections

STS_W3SVC#_#STS_W3SVC#_# - Site Collection data - Site Collection data containing documents, lists, surveys, containing documents, lists, surveys, discussions, webs, etc..discussions, webs, etc..

DatabaseBackup Optimization

Database sizesDatabase sizes

25-50GB Database Storage (>500 sites)25-50GB Database Storage (>500 sites)

Number of databases (>300)Number of databases (>300)

Cost SavingsCost SavingsDirect to Media?Direct to Media?

Backup compression/SpeedBackup compression/SpeedSQL LiteSpeed – 25-35% compressionSQL LiteSpeed – 25-35% compression

Snapshots or ClonesSnapshots or Clones

SharePoint Portal ToolsSPSBackup.exeFull Farm BackupFull Farm BackupSpsbackup.exeSpsbackup.exe /all /file <Backup path and file>/all /file <Backup path and file>

Central Portal Index Backup:Central Portal Index Backup:

Spsbackup.exeSpsbackup.exe /portal http://portal /service index /file \\server\share\backup /overwrite /portal http://portal /service index /file \\server\share\backup /overwrite

DatabasesDatabasesBackward-compatible Doc LibBackward-compatible Doc LibSPS IndexesSPS IndexesXML manifestXML manifest

schema versionschema versionportal GUID portal GUID Path to DB backup filePath to DB backup fileResultResult

Windows SharePoint Services ToolsSTSAdm.exe UtilitySTSAdm.exe Utility

stsadm.exe –o backup –url stsadm.exe –o backup –url http://team/sites/site -f file.bakhttp://team/sites/site -f file.bak

Site Collections OnlySite Collections Only

WSS sites – full fidelity w/ securityWSS sites – full fidelity w/ securityCustom assemblies/web parts not copiedCustom assemblies/web parts not copied

MySites and Team Sites on Portals – Not MySites and Team Sites on Portals – Not recommended by Supportrecommended by Support

Windows SharePoint Services Tools (Cont)Smigrate.exe “Smigrate.exe “The SharePoint Migration Tool”The SharePoint Migration Tool”smigrate –w http://vserver/sites/site -f c:\file.fwp smigrate –w http://vserver/sites/site -f c:\file.fwp

Site Collection & Sub Sites SupportedSite Collection & Sub Sites SupportedFiles & Lists copied to disk with manifest Files & Lists copied to disk with manifest Same as FrontPage2003 Client Backup/RestoreSame as FrontPage2003 Client Backup/RestoreSupport for Site Collections on SPS and Sub Support for Site Collections on SPS and Sub sites (not Areas), WSS, and Personal Sitessites (not Areas), WSS, and Personal Sites

** NOTE:** NOTE:My sites - No Personalization (Home Page My sites - No Personalization (Home Page reset), profile changes and Securityreset), profile changes and SecurityOther Sites and Site Collections –Security Lost Other Sites and Site Collections –Security Lost during backupduring backup

SharePoint BackupThird Party VendorsAvePointAvePoint

DocAve – SPS 2003 – Item level backup/restore/archive with DocAve – SPS 2003 – Item level backup/restore/archive with multiple migration options (supports SQL & Webstore)multiple migration options (supports SQL & Webstore)Downloadable Demo AvailableDownloadable Demo Available

Computer Associates (CA)Computer Associates (CA)BrightStor ARCserve Backup BrightStor ARCserve Backup

Backup/Restore SharePoint databases hosted on the various SQL Backup/Restore SharePoint databases hosted on the various SQL servers distributed across the enterprise.servers distributed across the enterprise.Advanced document level backup and restore supportAdvanced document level backup and restore support

CommVaultCommVaultGA, Galaxy- SPS 2003 database level backup and recovery GA, Galaxy- SPS 2003 database level backup and recovery productproduct

Enterprise wide backup and recovery of SPS 2003 data (SQL and Enterprise wide backup and recovery of SPS 2003 data (SQL and indices). Site and sub-site backup and recovery; Webstore backup indices). Site and sub-site backup and recovery; Webstore backup supportsupport

Product in Beta for “Doc Level” backup/restoreProduct in Beta for “Doc Level” backup/restoreVeritasVeritas

Backup Exec 10.0 (Late Q3 04) - SPS Server Level backupBackup Exec 10.0 (Late Q3 04) - SPS Server Level backup

SpbackupSpbackup

Automating Site Backup (Reskit Tool)Automating Site Backup (Reskit Tool)http://www.microsoft.com/technet/prodtechnol/sppt/reskit/c2861881x.mshttp://www.microsoft.com/technet/prodtechnol/sppt/reskit/c2861881x.mspxpx

Site ArchivalCleanup Storage

Auto Notification Auto Notification

Auto DeleteAuto Delete

Usage Logging and ReportingUsage Logging and Reporting

Analyze Site UsageAnalyze Site UsageLast ModifiedLast Modified

IIS usageIIS usage

Identify sites to “cleanup”Identify sites to “cleanup”

Archive/Backup then Delete targeted sitesArchive/Backup then Delete targeted sites

DBCC Shrink DBDBCC Shrink DB

Monitoring BackupsSharePoint Backup Relevant MOM Events

WSS MOM PackWSS MOM PackDatabase Capacity ReachedDatabase Capacity ReachedDatabase Warning ReachedDatabase Warning Reached

SPS MOM PackSPS MOM PackSPSBackup.exeSPSBackup.exe backup success/fail backup success/fail

Indexes or databaseIndexes or database

SQL Server 2000 MOM PackSQL Server 2000 MOM PackSQL Server database and transaction log space SQL Server database and transaction log space warnings and errors.warnings and errors.SQL Server failed backupsSQL Server failed backupsSQL Server failed agent jobsSQL Server failed agent jobs

Backup Media

Disk to TapeDisk to Tape

TapeBackup Optimization

Network Upgrade - Over NetworkNetwork Upgrade - Over NetworkPrivate NetworkPrivate NetworkSAN TapeSAN Tape

1GB/Min (db dump to tape)1GB/Min (db dump to tape)

Max drive or share sizeMax drive or share size100-300 GB for quicker restart100-300 GB for quicker restart

Script/Dev ideas

Incremental Site Backup – Incremental Site Backup – backup changed backup changed sitessites

Document backup – Document backup – Web folder backup of Web folder backup of documents or object model iterating through documents or object model iterating through documentsdocuments

Incremental Document backupIncremental Document backup – Backup – Backup changed documents using Object Modelchanged documents using Object Model

Archive – Archive – Use visits w/ modify date to Use visits w/ modify date to determine what sites should be archived determine what sites should be archived (backed up to disk) then deleted(backed up to disk) then deleted

Data Recovery

Disaster RecoveryDisaster Recovery

Recovery Scenarios & RequestsRecovery Scenarios & Requests

Recovery FlowRecovery FlowDatabaseDatabase

GranularGranular

Demo: Textcopy Demo: Textcopy Doc restore from Content DB Doc restore from Content DB

Portal RestorePortal Restore

Service LevelsService Levels

Disaster Recovery: Server

Web Front End – Web Front End – A. Restore failed drives and system state (metabase) A. Restore failed drives and system state (metabase) B. Reinstall, remove/re-add topology reconfigure, re-add web parts, B. Reinstall, remove/re-add topology reconfigure, re-add web parts, recreate virtual servers, extend and map to existing virtual serversrecreate virtual servers, extend and map to existing virtual servers

Search – Search – A. Restore failed drives and system state (metabase) propegateA. Restore failed drives and system state (metabase) propegateB. reinstall, remove/add back to topology and propagateB. reinstall, remove/add back to topology and propagate

Index – Index – A. Restore failed drives and system state (metabase) restore Index A. Restore failed drives and system state (metabase) restore Index using using spsbackup.exespsbackup.exereinstall, remove/add back to topology, restore indexes using reinstall, remove/add back to topology, restore indexes using spsbackup.exespsbackup.exe, and propagate, and propagate

SQLSQL – Restore failed databases (except Config db) – Restore failed databases (except Config db)Disconnect reconnect content databasesDisconnect reconnect content databases

Backward Compatible Doc LibBackward Compatible Doc Lib – –Disconnect Doc libs, remove from topology & farmDisconnect Doc libs, remove from topology & farmRe-add new server to farm & topologyRe-add new server to farm & topologyRestore using Restore using SPSBackup.exeSPSBackup.exe restore restore

Hot Swappable FarmSQL Log Shipping (See Reskit Chapter 28)

Mirror ServersMirror ServersSet DB Full or Bulk Logged recovery modeSet DB Full or Bulk Logged recovery modeMirror DataMirror Data

Full Backup of FarmFull Backup of FarmFull Restore of FarmFull Restore of Farm

Enable SQL Log shippingEnable SQL Log shippingSITE, SERV, PROF & Content dbs SITE, SERV, PROF & Content dbs (NOT CONFIG)(NOT CONFIG)

Set DBs on Mirror to Standby or No RecoverySet DBs on Mirror to Standby or No RecoveryConfigure FrequencyConfigure FrequencyCreate Farm, Customization/Portal & Virtual Server Create Farm, Customization/Portal & Virtual Server configconfig

Recommended: Periodic execution of “delete portals and Recommended: Periodic execution of “delete portals and detach databases; reconnect them to mirror farm config detach databases; reconnect them to mirror farm config DB” DB”

Recovery of Hot Swap Farm Via SQL Log ShippingSQLSQL

Close Open ConnectionsClose Open Connections

Change role of Secondary SQL to Primary (or Change role of Secondary SQL to Primary (or keep dbs read only until recovery of primary)keep dbs read only until recovery of primary)

SharePointSharePointAs needed Create Portal from DBs (connect)As needed Create Portal from DBs (connect)

Disconnect/Reconnect Content DbsDisconnect/Reconnect Content Dbs

WINS/DNS to point to new Farm VIPWINS/DNS to point to new Farm VIP

-Start a Full Crawl-Start a Full Crawl

Recovery ScenariosDatabase -Database - most reliable disaster recovery most reliable disaster recovery scenarioscenario

Site Collections – Site Collections – STSAdm.exe STSAdm.exe restorerestore

Database restoreDatabase restore

Document Library, List, Doc recovery Document Library, List, Doc recovery – – Database restoreDatabase restore

STSAdm.exeSTSAdm.exe restore restoreDocumentDocument- - Textcopy or other API can make retrieval Textcopy or other API can make retrieval from DB possiblefrom DB possible

Restore Requests

Default 15000 Sites MaxPer Content DB

Avg 20GB

>1% of Restore Requests

Includes Sub sites Workspaces

Avg 30-65 MB

41% Restores68/167 site68/167 site

Doc Libraries, list items, lists, events,Surveys, forms, etc..

Avg 1.5 MB

49% Restores82/167 doc82/167 doc

Database Restore

Tape to Disk to Online dbTape to Disk to Online db

Database Recovery

SITE / STS_W3SVC#_#SITE / STS_W3SVC#_# – restore database, – restore database, remove from config db using admin UI, re-remove from config db using admin UI, re-add using admin UIadd using admin UI

PROF DB – PROF DB – restore profile database, start full restore profile database, start full import, remove objects missing from import import, remove objects missing from import using UIusing UI

SERV DB – SERV DB – restore services database, bring restore services database, bring online (keep in sync w/ SITE DB)online (keep in sync w/ SITE DB)

Granular RecoveryPortal and below

Restore relevant databases (SITE, SERV, PROF, Restore relevant databases (SITE, SERV, PROF, STS_W3SVC#_#)STS_W3SVC#_#)

Restore to standby environment using Restore to standby environment using SPSBackup.exeSPSBackup.exe / /Restore tab GUIRestore tab GUI

STSAdmSTSAdm for site collection migration for site collection migration

SmigrateSmigrate for area or sub site migration (reapply security) for area or sub site migration (reapply security)

TextCopyTextCopy for doc restore for doc restore

Portal Restore From DBDBs Only (no index)

Central AdminCentral AdminClick - Create a portal siteClick - Create a portal site

Choose - Restore a portal siteChoose - Restore a portal site

Select Servers and databasesSelect Servers and databases

Choose Virtual Server and specify URLChoose Virtual Server and specify URL

Extend and MapExtend and Map

Granular RecoverySite/Sub siteSite CollectionSite Collection

Backed up Backed up Stsadm.exe –o restore –url Stsadm.exe –o restore –url http://http://sitenamesitename –f filename –f filename –overwrite–overwriteRequires 2Requires 2ndnd content db content db

No Site BackupNo Site BackupRestore content databaseRestore content databaseAttach database to Attach database to standbystandby virtual server/Lab virtual server/Lab environmentenvironmentBackup site using Backup site using stsadm.exestsadm.exeRestore using Restore using stsadm.exestsadm.exe with overwrite option with overwrite option

Sub site - use Sub site - use smigratesmigrate to move data to production to move data to production with overwritewith overwrite

Document Restore TextCopy

What Content Database to Restore?What Content Database to Restore?USE @WSS_configUSE @WSS_config

SELECT nameSELECT name

FROM sites s, databases dFROM sites s, databases d

WHERE s.databaseid = d.databaseidWHERE s.databaseid = d.databaseid

AND urlpath = @urlpathAND urlpath = @urlpath

VariablesVariables@WSS_config = configuration database name@WSS_config = configuration database name

@URLpath=Site; @leafname = filename @dirname = directory name of @URLpath=Site; @leafname = filename @dirname = directory name of file file

SQL Query to Define DocWhat Site and Document?What Site and Document?USE [X1_SITE]USE [X1_SITE]SELECT content SELECT content FROM docsFROM docsWHERE dirname = @dirnameWHERE dirname = @dirnameAND leafname = @leafnameAND leafname = @leafname

Example Text CopyExample Text Copytextcopy /s server /u @user /P @password /d textcopy /s server /u @user /P @password /d

“@database" /t docs /c content /W "where “@database" /t docs /c content /W "where leafname= ‘@leafname’ and dirname= leafname= ‘@leafname’ and dirname= ‘@dirname’” /F c:\temp\leafname /O /Z‘@dirname’” /F c:\temp\leafname /O /Z

VariablesVariables@WSS_config = configuration database name @database = content database@WSS_config = configuration database name @database = content database@URLpath=Site; @leafname = filename @dirname = directory name of file @URLpath=Site; @leafname = filename @dirname = directory name of file

IT ImprovementsTapeTape

SAN Tape/Fiber Attached BackupSAN Tape/Fiber Attached BackupThroughput 1.2 GB/Min, 2-3xThroughput 1.2 GB/Min, 2-3x

Split drive into ~300GB shares for Tape BackupSplit drive into ~300GB shares for Tape BackupGB Network upgrade on SQL boxesGB Network upgrade on SQL boxes

DatabaseDatabaseReduce database sizes Max 25-50GBReduce database sizes Max 25-50GBReduce DB to >300DB per SQL instanceReduce DB to >300DB per SQL instanceMulti backup and SQL LiteSpeedMulti backup and SQL LiteSpeedSPSBackup.exeSPSBackup.exe to backup indexes only to backup indexes onlyDefragment Drives and databasesDefragment Drives and databases

Site CollectionSite CollectionIncremental Backups (backup changed sites)Incremental Backups (backup changed sites)Cleanup/AutoDelete – ArchiveCleanup/AutoDelete – ArchiveSite Backup storage on Passive Node/File serverSite Backup storage on Passive Node/File serverGB Network upgrade on Web ServersGB Network upgrade on Web Servers

Restore process flow better understoodRestore process flow better understood

+IT SharePoint Best Practices

Limit # of site collections per database to Limit # of site collections per database to 250-500250-500

Limit Large sites at ~2GBLimit Large sites at ~2GB

Use few IIS Virtual ServersUse few IIS Virtual Servers

Consolidate app pools for common appsConsolidate app pools for common apps

Blocked File TypesBlocked File Types

Antivirus on SQL Servers: Exclude FTS, Antivirus on SQL Servers: Exclude FTS, Database & Log drivesDatabase & Log drives

Additional Lessons Learned

4 GB RAM on Web Servers use /3GB in 4 GB RAM on Web Servers use /3GB in boot.iniboot.ini-g512 startup parameter in SQL for -g512 startup parameter in SQL for additional MTL (Mem To Leave)additional MTL (Mem To Leave)What to watch: App Pools & SQL Memory, What to watch: App Pools & SQL Memory, NIC, SQL Server Connections, SQL NIC, SQL Server Connections, SQL Blocking, disk usageBlocking, disk usage# of databases below 600# of databases below 600Isolate Critical Apps in their own app poolIsolate Critical Apps in their own app poolWatch for w3wp.exe memory exceeding Watch for w3wp.exe memory exceeding 800mb800mb

Q&A

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Q1: Overall satisfaction with the session

Q2: Usefulness of the information

Q3: Presenter’s knowledge of the subject

Q4: Presenter’s presentation skills

Q5: Effectiveness of the presentation

Please fill out a session evaluation on CommNetPlease fill out a session evaluation on CommNet

Attend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp

List of newsgroupshttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspx

MS Community Siteshttp://www.microsoft.com/communities/default.mspx

Locate Local User Groupshttp://www.microsoft.com/communities/usergroups/default.mspx

Community siteshttp://www.microsoft.com/communities/related/default.mspx

Three ways to access Online Evaluation Forms:Three ways to access Online Evaluation Forms:

MGB Evaluation Kiosks MGB Evaluation Kiosks

CommNet CommNet

From your laptop or Pocket PC, From your laptop or Pocket PC, onsite Wireless Access or any wired connection to: onsite Wireless Access or any wired connection to: http://mgb2004.com/evalhttp://mgb2004.com/eval

Please Complete An Evaluation FormYour input is important!

For more information please refer to the Registration GuideFor more information please refer to the Registration Guide

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.