Zero Data LossRecoveryAppliance - Oracle · •Use Recommended RA Software to avoid known critical issues –Number 1 method to avoid problems: Upgrade to recommended software release

ZeroDataLoss Recovery Appliance:Maintenance&OperationalBestPractices

MAABestPracticesTeamServerTechnologiesJuly2019

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.

SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

3


Agenda

4

UnderstandRequirementsFirst

RADeploymentDecisionsandKeyPractices

RAStayHealthyPlan

BestPracticesBackupandRestore

MigrationwithZDLRA

OracleSupport&ServiceRequest

RARolesandResponsibilities

1

2

3

4

5

6

7

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved. 5

MAARecommendations

• OneRecoveryAppliance(RA)perdatacenter

• BackupprimaryandstandbydatabasestotheirrespectivelocalRA

• NoRAreplicationforanydatabasewitharemotestandby

• RestoreoperationcanuseanyRAinanylocation


DoyourbusinessrequirementscallformultipleRecoveryAppliances?

1. Doyouneedtheabilitytobackup,restoreandrecoverduringRAplannedmaintenancewindows(2-6hoursformajorupgrades)versuswaitinguntiltheplannedmaintenancecompletes?

2. Doyourequiredisasterrecoveryprotection?(e.g.datacenterfailure,powerfailure)

3. DoyourequireminimaldatalossforallprotecteddatabasesbackinguptoRAduringitsplannedmaintenancewindows?(youdon'tneedtorestore/recover)

4. DoyourequirearchiveandbackupsoperationstocontinueduringRAplannedmaintenancewindows(2-6hours)versuswaitinguntiltheplannedmaintenancecompletes?

If youanswered“Yes” ,toanyofthesequestionsmultipleRecoveryAppliancesarerequired.

Yes/No

Yes/No

Yes/No

Yes/No


MultipleRecoveryAppliancesRequired?

q Didyouanswer“yes”?

ü ToANYofthequestions:YouwillneedanotherRecoveryAppliance

ü Yestoquestion#1or#2:UseHighAvailabilityforBackupandRecovery

ü Yestoquestion#3or#4Only:UseBackupFailovertoAlternateApplianceSolution

Note:UsingDataGuardorGoldenGate?Backuptheproduction,standbyorGoldenGatereplicatotheirrespectivelocalRA.


TapeLibrary

OneWay

Bi-Directional

Hub&Spoke

RemoteDataCenterLocalDataCenter

RAReplica:SystemOutage/DisasterProtection

REPLICATIONBENEFITS• ProtectsdatafromLocal

orSite-widefailures• Automatedrestorefrom

LocalApplianceordirectlyfromRemoteAppliance

8

RecommendedforStandalone(NonDataGuard)Databases


RAOsakanormallyreplicatestoRATokyoWhenupstreamappliance(RAOsaka)isnotavailable,backupsandredoareredirectedtoRemoteappliance(RATokyo)

• Virtualfullsarecreatedasnormal

• SizingDRapplianceIngeneral,sizeperRecoveryWindowGoal(RWG)businessrequirement:1xfullbackup+NRWGdaysofincrementalandredo/archlogbackupsBareminimum:1xfullbackup+1dayredo/archlogsbackups.

Whenupstreamisbackonline,DRappliancebackupsaretransferred

• Backupsareingestedandprocessedintovirtualfulls

• Normalbackupstoupstreamcanberestartedimmediately

• Virtualfullsfornewbackupsarecreatedafteralltransferredbackupshavecompletedprocessing

• Note:RedoLogssenttothedownstreamarenotautomaticallycopiedtotheupstream.Arestore/recoverwillpullthelogsfromthedownstreamasnecessary.

Benefits

• BestpracticetopreserveHAduringplannedorunplanneddowntime

• Databasebackup&restore/recoverabilityavailablefromeitherupstreamanddownstream

9

X

HighAvailabilityforBackup&Recovery

RAOsaka

Replication

RAOsaka RATokyo

RATokyo

Replication

RAOsaka RATokyo

BackupstoDRAppliance

DRAppliancebackupstransferredtoUpstream


BackupFailovertoAlternateRA• IncrementalsandRedonormallysenttoPrimaryRA

• AlternateRAservesasbackupstagingareawhenprimaryRAisunavailable,thensyncswithprimaryRAafterwards– Novirtualfullscreatedonalternate,hencerecoverabilitynotsupported

– Spacesizedfor‘n’incrementalsandarchivedlogbackupsduringprimarydowntimeperiod

• Benefits:– Preservesbackupandredoshipmentcontinuityduringplannedmaintenance/upgrades

– PreventslocalFastRecoveryAreasfromfillingupwitharchivedlogs

– BlockChangeTrackingcontinues

10

RAOsakaPrimaryAppliance

XRATokyoAlternateAppliance

Alternateappliancetakesoverbackupsandredotransport

Incrementals&

Redo/Arch

“BF_FORWARD”PolicyonAlternateRA:STORE_AND_FORWARD=‘YES’


BackupFailovertoAlternateRA- Continue• Backupandredofailoverdestinationwhenprimaryapplianceisunavailable

– Alternateappliancetakesoverbackupsandredotransport

– Whenprimaryapplianceisbackonline,allinterimbackupsarereplicatedfromalternateandvirtualfullbackupsarecreatedonprimary

– Onceallvirtualfullsarecompleted,backupsandredotransportcanrestarttotheprimaryappliance

– BackupFailoverwithinthesamedatacenterrequiresbothRecoveryAppliancestobeconfiguredonthesamereplicationsubnet.

11

“BF_FORWARD”PolicyonAlternateRA:STORE_AND_FORWARD=‘YES’

RAOsakaPrimaryAppliance

BFOSAKA

“BF_BACKUP”PolicyonPrimaryRA

RATokyoAlternateAppliance


RecoveryApplianceSolutionsSummary

– HighAvailabilityforBackupandRecovery(Non-DataGuardDatabases)• BackupandRedotoRAReplicaduringRAmaintenanceandunplanneddowntime• RestorefromRAReplica• MorecapacityrequiredduetoRAreplication• See:ConfiguringHighAvailabilityZDLRAclientforbackupandrestore(DocID2432144.1)

– BackupFailovertoAlternateAppliance(Standalone/Primary/StandbyDatabases)• BackupsandRedotoAlternateRAduringRAmaintenanceandunplanneddowntime• RestorefromAlternateNOTavailable• LesscapacityrequiredbecauseRAreplicationwasnotsetup• See:ImplementingAdditionalHighAvailabilityStrategies intheZeroDataLossRecoveryApplianceAdministrator'sGuide.

12


RecoveryAppliance&DataGuard

PrimaryDatabase

RemoteStandbyDatabase

DataGuardTransportsync/async

FastIncremental FastIncremental

Redo(ASYNC)

Redo(ASYNC)

MAARecommendations:• OneRecoveryAppliance(RA)perdatacenter• BackupprimaryandstandbydatabasestothelocalRA• NoRAreplicationforanydatabaseswitharemotestandbyneeded• RestoreoperationcanuseanyRAinanylocation

StandbyLocationPrimaryLocation

Fastincremental(anincrementalusingBlockChangeTracking)onastandbydatabaserequiresanActiveDataGuardLicense.

RAReal-TimeRedodoesNOTrequireanActiveDataGuardLicense.


RecoveryApplianceandDataGuard

• PostDataGuardroletransition– Nochangeinbackupoperations.ContinuetobackupboththeprimaryandstandbydatabasestothelocalRA

• DeployingtheZeroDataLossRecoveryApplianceinaDataGuardConfiguration– Refertohttp://www.oracle.com/technetwork/database/availability/recovery-appliance-data-guard-2767512.pdf / DeployingZeroDataLossRecoveryApplianceinaDataGuardConfiguration (UpdatedMarch2018)

MaximumAvailabilityArchitecture(MAA)WhitePaper

14


Agenda

15

Understand Requirements First

RADeploymentDecisionsandKeyPractices

RAStayHealthyPlan


MigrationwithZDLRA



1

2

3

4

5

6

7


NetworkConfigurationOptions

• ZDLRASupports10GigE/25GigEandInfiniBand– 10GigEor25GigEistherecommendedforprotecteddatabasetoRAconnectivity• EthernetbasedconnectivityprovidesoptimalbackupandrestoreratesduetoRA’sresourcemanagement.

• Note:WhileIBconnectivityissupported,beawareofthefollowing:– IBwillnotrealizethefullIBnetworkbandwidthduetoRAresourcemanagement(managedthrottling).– IBsetupwillintroducesoftwareupdatecomplexity.– Owner’sGuide,Chapter9hasdetailsonhowtoconfigureBackupIngestoverIB

– Note:Real-TimeRedoTransportuses10GigEor25GigEnetworkONLY

16


NetworkConfigurationOptions- continue• VLANfornetworkisolation– BackupandrestoretrafficfromdifferentVLANsisnotrouted.– RecoveryAppliancesupportsVLANtaggingontheingestnetwork• ProtectedDBhostsondifferentandisolatedVLANscanbeconnecteddirectlytotheRA• Enabling8021.QVLANTagginginZeroDataLossRecoveryApplianceOverIngestNetworks(DocID2047411.1)

• ReplicationNetwork:TypicallyconfiguredviaOEDAbeforeRASWinstall– Replicationnetworkconfiguration,postinstall• PostInstall- ReplicationNetworkConfigurationforZDLRA(DocID2126047.1)

–WhenReplicationnetworkisconfigured:VLANtaggingisnotsupported.

– ProtectedDatabasesmustusetheingestnetwork.

17


RecoveryApplianceSecurity

• ClienttoRecoveryAppliance,orRecoveryAppliancetoClient– Upcoming:SecurityinFlight(TCPS,HTTPS)è Provideduponrequest– AddingaprotecteddatabaseRecoveryApplianceVPCusercredentialstoanexistingOIDwallet(DocID2211759.1)

• SecurityintheRecoveryAppliance– RecoveryApplianceadministratorsresponsibilities• CreateVirtualPrivateCatalog(VPC)User• AssignprotecteddatabasestoaspecificVPCUser• TheprotecteddatabaseadministratorcanseealldatabasesthatshareacommonVPCuser

Customersrequiringendtoendsecurity

18


Agenda

19


RA DeploymentDecisionsandKeyPractices

RAStayHealthyPlan


MigrationwithZDLRA



1

2

3

4

5

6

7


AdjustingDefaultSettings• DonotmakeanychangestoRecoveryAppliance– IT’SANAPPLIANCEandit’salreadyoptimized– RestrictionsandsupportedconfigurationexceptionsinMOSnote2172842.1

• Ifrequired,setMAX_RETENTION_WINDOW toenforcehardlimitsondataretentionforalldatabaseswithinaprotectionpolicy.– E.g.Usecaseswheretherearestringentcompliance/regulatoryrequirements– Backupsareforciblyremovedafterexceedingwindowviapurgeimmediatetasks– Startingin12.2.1.1.2NovemberPSU,MAX_RETENTION_WINDOWissetto2XRecoveryWindowGoalbydefault.• Adjustbasedonbusinessrequiredmaxretention,ifneeded– mustbe>RecoveryWindowGoal

20


UseRecommendedRASoftware• SubscribetoMOSalertsandreferperiodicallytothefollowingnotes:

• RecoveryApplianceCriticalIssuesMOSnoteforcriticalissuesalerts (SUBSCRIBENOW)• RecoveryApplianceSupportedVersionsMOSnoteforlatestsoftwareupdate

• UseRecommendedRASoftwaretoavoidknowncriticalissues– Number1methodtoavoidproblems:Upgradetorecommendedsoftwarerelease– ZeroDataLossRecoveryApplianceSupportedVersions(DocID1927416.1)– ZeroDataLossRecoveryApplianceUpgradeandPatching(DocID2028931.1)– Patchesarecumulativeandinclude

- Bugfixes(Avoidabugbeforeithappens)- Enhancements(ModificationtoRAprocessflow)

• CoordinatewithPlatinumPatching– Scheduleearly&OpenProactiveSR

21

VeryImportant


Don’ts

– Don’tignoreincidents:• Resolveandunderstand

– Don’tneglecttheRA• Monitorthesystem

– Don’tSubmitmultiplerequesttoDeletedatabases• Multiple– concurrentdatabasedeletionscanhaveanegativeimpactontheRA• AvoidDeleteDatabasecommandsasmuchaspossibleorlimitdeletionsto1or2concurrentlyforlargedatabasedeletions• Startwiththesmallestandworkup,monitorforsuccessfulcompletion.

22


Don’ts– Don’trepeatbackuporrecoveryscriptsafterafailure• Firsttroubleshootthereason.• Typicallyaclientsideissue.• RapidandrepeatedexecutionsofascriptmayimpacttheRA.

– Don’tmakemodificationstotheRAconfiguration• ZeroDataLossRecoveryAppliance- InstallingThird-PartySoftwareandModifyingInternalApplianceSoftware(DocID2014361.1)- (oratab,.zdlra env,databasemachine.xml)

– Don’ttakeperiodicLevel0• VirtualLevel0requiresonlyonelevel0followedbylevel1s• Note:Alevel0mayberequested– BySupport– TopreserverecoverabilityineventofbackupcorruptiondetectedontheRA

23


Don’ts– Don’tbackupaDataGuardprimary& standbytothesameRA• SeeDeployingtheZeroDataLossRecoveryApplianceinaDataGuardConfiguration

– Don’tregister databaseswithidenticalDBIDstothesameRA• SeeCloningaProtectedDatabase”withintheZDLRAProtectedDatabaseConfigurationGuide

– Don’tbackuptoanothermedia• SwitchingtoanothermediacanimpactpastandfuturebackupsmadetotheRA

– Dualbackupstrategiesarecomplexandshouldbeavoidedifpossible• Dualbackupstrategiesaredesignedformigration• DualBackupStrategyMOSNotes:– ImplementingaDualBackupStrategywithBackupstoDiskandRecoveryAppliance(DocID2154461.1)– ImplementingaDualBackupStrategywithBackupstoTapeandRecoveryAppliance(DocID2154471.1)

24


Dos

– Validatethebackup/restorenetworkfirst:• UseZeroDataLossRecoveryApplianceNetworkTestThroughputscript(DocID2022086.1)– requiresqperf,NOTOSagnostic

• HowtomeasurenetworkperformancefromRMANforZDLRAorCloudBackups(DocID2371860.1)– usesRMAN’s“NETTEST”option,OSagnostic

– FileaNEWSRforanynewissue• RefertoServiceRequestsandEscalationProcesssection

– SetNLS_DATE_FORMATbeforecallingrmanscripts• e.g.inLinux:exportNLS_DATE_FORMAT="yyyy-mm-dd hh24:mi:ss"

25


Dos

– UseMultiSection:setSECTIONSIZEto64GB• Numberofsectionsperdatafileislimitedto256.• Whenasectionsizeof64GBisused:– Largedatafiles>16TBwillautomaticallysee>64GBsectionsizeasdeterminedbyRMAN• if(sizeof(datafile)>16TB),sectionsize=sizeof(datafile)/256

– Smalldatafiles<64GBwillnothavesections• A64GBsectionsize:allowsforefficientprocessinginZDLRA’sflashcache– ForcesFILESPERSETto1

– Usethelatestlibra (TheRMANclientsbtlibrarythatsupportsZDLRA)• Downloadthelatestsbt library (libra module)(DocID2219812.1)• ForRAC:DistributetoALLnodesinthecluster• DoNOTupdatethelibra.soontheZDLRA,unlessdirectedbysupport.

26


MonitoringRA’sHealth

• MonitortheApplianceonadailybasis– EMUnifiedManagementDashboard• Reviewtwicedaily

– SystemActivityScript(DocID2275176.1)• Rundailyandmonitortrends

– EMnotifications• Reviewandactonnotifications

– RunexachkMonthlyandreviewfindings• HowToupdateexachkoutsideZDLRAInstall,PatchingandUpgrade(DocID2399688.1)• Usedifftocomparemonthtomonth• Runpreandpostpatching

– ReviewCapacityPlanningReportMonthlyorBi-Monthly

27


IstheZDLRAHealthy?OracleEnterpriseManager

28


UnifiedEnd-to-EndControl

• RecoveryApplianceAdmincentrallymonitorsandmanagesalldatabaseprotectionactivityacrossalltiers

• DatabaseAdminmonitorstheprotectionstatusoftheirdatabasefromdisk,totape,toreplica–OffloadedreplicasandtapebackupsappearinRecoveryCatalog

Tape

RemoteAppliance

Enterprise Manager

Tape

29


BestPractice– UnifiedManagementDashboard– 1)Summary

• Showsthenumberofprotecteddatabases,andsummarizestheirhealthstatus,currentactivity,andactivitywithinthelast24hours.Formoreinformation,clickthelinksintheOperationcolumn:Backup,Copy-to-Tape,Replication,andRestore.

– 2)ProtectedDatabaseIssues• Highlightsissuesrelatingtobackupandrecoverystatusforprotecteddatabases.TheViewmenufiltersthedataonkeycategories.

– 3)DataSent/Received(Daily)• Displaysdailythroughputoverthepastweek.

– 4)Performance• ChartsperformancestatisticsforDataRateandQueuedData.Thestatisticsarefilterablebyday,week,ormonth.

– 5)MediaManagers• Displaystheconfiguredmediamanagerforcopy-to-tapeoperations.

– 6)StorageLocations• Summarizestotalavailablespaceandusagebyindicatinghowmuchhasbeenconsumedtomeetthediskrecoverywindowgoalforall

databases,andwhatpercentageoftotalspaceisreservedspacefordatabasesbackinguptothespecifiedstoragelocation.

– 7)Replication• ListsthedownstreamRecoveryAppliancestowhichthisRecoveryApplianceisreplicating,andalsotheupstreamRecoveryAppliances from

whichthisRecoveryApplianceisreceiving

– 8)IncidentsandEvents• SummarizesallwarningsoralertsthathavebeengeneratedbyCloudControlmonitoringofalltargetsassociatedwiththeRecoveryAppliance.

Fromthissection,drilldownforfurtherdetailontheissues.

30


BestPractices– UnifiedManagementDashboard- Continue

1

2

3

4

6

5

7

8


BestPractices– LeverageOEMnotifications

RecoveryAppliancealertnotificationsetupusingOracleEnterpriseManager(DocID2262003.1)

Exampleofalertsandnotificationforoldbackups:


ZeroDataLossRecoveryApplianceSystemActivityScript• ZeroDataLossRecoveryApplianceSystemActivityScript(DocID2275176.1)

• Containsmultiplequeries,providing:– Catalogversion– Generalstateofthesystem– Examinationofrunningtasksonthesystem– Taskhistoryforthelastday– Spaceusage– Lockinginformation– Checkstatusofreplicationserverifitexists– Incidentsforthelastfivedays– Displayeachdatabase'scurrentprogressprocessingtheirdatafiles– APIcommandsoverthelast2weeks– Notableconfigchanges

33


IstheZDLRAHealthy?SystemActivityReport

34

• Itemstowatchfor:BadreportWatchfor:

IftherearetasksofWORKTYPEinRUNNINGstateandcreatedadayearliertheninvestigate.

IftherearetasksofMAINTENANCEorSBTTYPEpresentandtheircreationtimeisolderthanoneweekthenitshouldbeinvestigated.

Alargenumberoftaskinexecutablestateforthesametask_type.

Thesametasktypeisincreasinginthenumberofjobs(current_count)overtime.Asystemwithorderingwaitsthatareolderthan1day.

Asystemwithtasksinstall_when_wait.ThisshouldonlybeseenifOracleSupportistroubleshootingtheRA.


IstheZDLRAHealthy?SystemActivityReport

35

TheGood:

Minimumcreationtimeforactivetasksiswithinthelast24hoursforworktasks.

Taskhistorystateshouldindicateworkisbeingcompleted.

Reviewothersections


IstheZDLRAHealthy?exachk

36

UpdateexachkforRA:• HowtoupdateexachkoutsideZDLRAInstall,Patching

andUpgrade(DocID2399688.1)


exachk&RAInformationCenter

InformationCenter:OverviewZeroDataLossRecoveryAppliance(DocID1683791.2)


HowtoDetermineIfZDLRAIsKeepingupWithLoad• Determiningwhetherunprocessedtasklistisgrowing(over7dayperiod):

SELECT count(*) FROM ra_task

WHERE archived=‘N’;

• Determinewhetherincidentsareactivefordelaysinperformingbusywork:SELECT error_text

FROM ra_incident_log WHERE status=‘ACTIVE’

AND error_text like '%has not run%';

• Taskswithhighestnumbers(RA_TASK.PRIORITY) arebusyworktasks– DB_STATS_REFRESH,RESTORE_RANGE_REFRESH,OPT_DF,OPTIMIZE,REBUILD_INDEX,VALIDATE,CHECK_FILES,CROSSCHECK_DB

38

Problem:taskqueuegrowingovertimeAction:reviewsystemactivityreport/EM

openSR

Problem:ActiveincidentslogindicatesbusyworknotrunningAction:Reviewsystemactivityreport/EM

openSR


ORDERING_WAITtasks

39

• Occurswhenabackuppiecewon’ttilewithexistingDeltaPools• Causes:• Unprocessedincrementalsfrompollingdirectory• Missingincrementallevel1backup• Incremental level1backuptakenagainst alevel0backupthatisnotinDeltaPool(maybeonlocalstorageinProtectedDBorreplicaorontape• Missingdatafileincarnationinformation,e.g.OPENRESETLOGSfollowedbymissingresync• Unsupportedfeaturesusedinbackuppiece:

MAXPIECESIZE,RMANencryption


ORDERING_WAITtasks-- investigating

• CurrentscriptforinvestigatingORDERING_WAITissuesisavailablefromsupportinaMOSnote:– DiagnosticSQLscriptfortasksinORDERING_WAITstatusonRecoveryAppliance(DocID2095949.1)

40


LostRESTORE_RANGE

• Usesofrestoreranges:– RECOVERYRANGEoutputfromEM– RA_<disk|sbt>_RESTORE_RANGEviews– UNPROTECTED_WINDOWinformationandalerts

• Forarestorerangetobevalidweneed:– Archivedlogstocovertherange– ZDLRAbackupsforalldatafilesthatweretakenduringtherange– Backupofcontrolfile

41


DiagnosingRESTORERANGEproblems• ComparerestorerangeofZDLRAstorageagainstallstorage

select*fromra_disk_restore_rangewheredb_key=<db_key>orderbylow_time;------------------------------select*fromrc_restore_rangewheredb_key=<db_key>orderbylow_time;

• Makesurethattherestorerangehasbeenrecomputed

selectcount(*),max(completion_time)fromra_taskwheretask_type=‘RESTORE_RANGE_REFRESH’andarchived=‘Y’anddb_key=<db_key>;

• SeewhatRMANthinksaboutrecoveryofdatabase

RMAN>restorepreviewdatabase;

42


DiagnosingRESTORERANGEproblems(cont.)• Getlistoflogstomakesurethattheytile

selectthread#,first_change#,first_time,next_change#,next_timefromrc_backup_redologjoinrc_backup_pieceusing(bs_key,db_key)wheredb_key=<db_key>andba_access='Local'andfirst_time>sysdate - <RWG>orderbythread#,first_change#;

• Getlistofdatafilestomakesurethattheyoverlapwitharchivelogsselectfile#,checkpoint_change#,checkpoint_time,absolute_fuzzy_change#fromrc_backup_datafilejoinrc_backup_pieceusing(bs_key,db_key)wheredb_key=<db_key>andba_access='Local'orderbyfile#,checkpoint_change#;

• Getlistofcontrolfilestomakesurethatoneisusableselectc.completion_timefromrc_backup_controlfilecjoinrc_backup_pieceusing(bs_key,db_key)wheredb_key=<db_key>andba_access='Local'orderbyc.completion_time;

43


RecoveryApplianceBIReports• ReportsareavailabletohelptheRecoveryApplianceadministratorunderstandresourceutilization,alerts,andhistoricaloperations• Reportsexamplesinclude:– ActiveIncidents– APIHistory– CapacityPlanning– RecoveryWindowSummary– Top10DatabasesbyDataTransfer– ProtectedDatabasesDetails– ChargebackReports

44


ProtectedDatabaseReportRecoverabilityinformationperdatabase!

Topsectioncontainsthefollowinginformation:

- ProtectedDatabase- SpaceUsed- RPO/RecoveryWindow- UnprotectedWindow- LastBackuptime- LastTapeCopy- LastReplication


ChargebackReport– PayasRAStorageisUtilized(Least)

• Scenario:Thinkofthisassimilartoanmeteredmodelwherethecustomeronlypaysforutilization.Thedatabasebeingprotectedischargedonlyforthespaceutilized.

• Example:A6monthretentionwouldrampupincostsinceutilizationonmonth1islowerthanmonth6.

46

ChargeforspaceasitisusedontheRecoveryAppliance


ChargebackReport- PayasRAStorageisUtilized-cont’d

• MonthlyRAStoragespaceconsumptionforthisdatabase.

• DisplayofRAspaceusedtocalculatethechargeback.

47



• MonthlycostforthisdatabaseonRAStorage.

48



• Monthlytapestoragespaceconsumptionforthisdatabase.

49



• Monthlycostforthisdatabaseontape.

50


Agenda

51



RAStayHealthyPlan


MigrationwithZDLRA



1

2

3

4

5

6

7


Backup&ConfigurationOverview

• StepstoBackup&ConfigureaDatabaseusingEnterpriseManagerCloudControl– CreateProtectionPolicyonRecoveryAppliance(RA)– AddProtectedDatabasetoRA– ConfigureBackupSettingsforProtectedDatabase– ScheduleONE-TIME Level0(Full)Backupwith“CustomBackup”,thenLevel1swith“Oracle-SuggestedRecoveryApplianceBackup”


WhatToDoWithExistingBackups?

• RMANbackupstodiskorNFSshare(includingDataDomainshare)– CanbeimportedintotheRecoveryAppliancevia“polling”– BecomepartoftheIncrementalForeverstrategyontheRA

• Backupstakenusing3rd partybackupsoftware– LeavetheagentinplaceontheprotectedDBhostsuntilretentionexpires– Leveragedualbackup(disk &tape)strategyifneededininterim– Removingagentssavessystemresources

53


BackupBestPractices• UseTransparentDataEncryption(TDE)insteadofRMANencryption– RMANencryptionwillpreventZDLRAfromcreatingVirtualFullBackups(VB$).

• UsenativedatabasecompressioninsteadofRMANcompression– RMANcompressionwillresultinadditionalCPUutilizationontheProtectedDatabaseclient– RMANcompressedbackupsrequiredecompressionandrecompressionontheZDLRA

• Useblockchangetrackingforallprotecteddatabases

54

rman target<targetstring>catalog<catalogstring>backupdevicetypesbtcumulative incrementallevel1filesperset 1sectionsize64gdatabaseplusarchivelognotbackedupfilesperset 32;


ImpactofTDETablespace EncryptionontheZDLRA• AdvancedSecurityOptionrequiresaLevel0when:– youfirstkey(encrypt)atablespace– yourekeyatablespace

FortheRA:• UpgradetoRA12.2.1.1.2-201810-RELEASE(INTERIM1)OCT2018PSUorlater• Whenyoufirstkey(encrypt)orrekeyatablespace:– TheNEXT backupforthattablespace MUST bealevel0

• RMANcompressionagainstaTDEtablespace willresultinRMANencryption,preventingtheRAfromcreatingVirtualBackups.

55


BackupBestPractices– Cont.• Incrementallevel0(full)asfirstbackup• Subsequentcumulativeincrementallevel1backups“IncrementalForever”• Usesectionsize of64GB• Usefilesperset 1foronedatafileperbackupset• Limitchannelallocationto4pernode• VirtualFullBackupCreationMonitoring– AfteraL1incrementalbackup:RAindexesitandbuildsthecorrespondingVirtualFull.CheckEMorSARforerrormessageslike“ORA-64760:DatabaseXYZhashadtasksinorderingwaitstateforoverXdays.”• RefertoMOSnote: DiagnosticSQLscriptfortasksinORDERING_WAITstatusonRecoveryAppliance(DocID2095949.1)

56


Restore&RecoveryBestPractices• UseRMANRestoreDatabase/RecoverDatabaseasyouwouldtoday– NonewRMANcommandstolearn.Intelligentbuilt-inrecoverycataloginRA– RMANisawareofthevalidatedbackupsondisk,tapeorreplica.Restoreistransparentandsimple– RestoredirectlyfromtapeorRAReplicawithoutstagingonlocalRAiflocaldiskbackupsarenotpresent

• Performanceconsiderations– Maximize#ofRMANchannelsforRestoreunlesstheyareotheractivedatabasesonthetarget– RestoreoperationsarealwaysautoprioritizedwithinRAwithoutpreventingotherbackupoperations– BCT:Continuebenefituntilallslotsareused.Defaultis7days.

• BigfileTablespacePracticesandConsiderations(recallbackupbestpracticeusingSECTIONSIZE)– OracleDatabase11gcanrestoreinitialL0withSECTIONSIZEtoparallelizesectionsacrosschannels• Restoringvirtualfulls(createdfromL1s)does parallelizesections(RA12.1.1.1.8.201710PSUandlater)

– OracleDatabase12ccanrestoreL0andvirtualfullswithSECTIONSIZEparallelismacrosschannels

57


Agenda

58



RAStayHealthyPlan


MigrationwithZDLRA



1

2

3

4

5

6

7


DatabaseMigrationProcessusingZDLRA

• CrossEndian- BigEndianó LittleEndianUsesVirtualLevel0backupofsourcedatabaseandRestore/RecoverondestinationRefertoMOSNote:CrossPlatformDatabaseMigrationusingZDLRA(DocID2460552.1)

• SameEndian- LeverageDataGuardDataGuardSupportforHeterogeneousPrimaryandPhysicalStandbyinSameDataGuardConfiguration(DocID413484.1)CreatingaPhysicalStandbyDatabaseinan11.2,12.1,12.2orlaterenvironment(DocID2275154.1)1

SourcedatabasebackupsavailableontheZDLRA,instantiatenewdatabaseonthedestinationplatformusingRMANDUPLICATEFORSTANDBY1WhenusingRMANtoDUPLICATEadatabase,connecttotheZDLRAasCATALOGandconfiguretheRMANAUXILIARYSBTchannelsfortheZDLRA.

59


CrossPlatformDBMigrationusingZDLRA- Benefits• Primarybenefits:– Significantreductionindowntime(Potentiallylessthan2hoursofservicedowntime)– ApplicationserviceisReadOnly duringmostoftheservicedowntime– NewZDLRAtool(dbmigusera.pl)simplifiescrossplatformmigrationbyautomatingsteps,especiallyusefulforlargedatabases–Migrationtimeisnotimpactedbythesizeofthedatabase– SupportsmigrationtothesameorhigherDBversion

ReadOnlydowntimebasedon– Finalincrementalbackup&recoverystepIncrementalsize<5%ofthedatabase– Finaltablespace metadataexport/importduration(onlynecessaryforcrossplatform)– Numberoftargettablespaces – importisdoneserially– Applicationserviceswitchoverfromsourcetodestinationdatabase

60


LeverageDataGuardandZDLRA- Benefits

• Primarybenefits:

– Near-ZerotoZerodowntimeduringswitchovertodestinationdatabase– Increasedavailabilityduringthemigrationprocessvstransportabletablespaceapproach–WithActiveDataGuard,standbydatabasecanbeusedtooffloadread-onlyactivitiesfromprimary(e.g.reporting),untilswitchoverisperformed

61


Agenda

62



RAStayHealthyPlan


MigrationwithZDLRA



1

2

3

4

5

6

7


OpeningaServiceRequest• ProactivelyupdatetheSRwithasMUCHdetailsasyoucan

***criticaltime-sensitiveinformationmightbelost!***

1. ProblemstatementwithUseCase,Timeline(beforeproblem,anychanges,whenproblemoccurs)2. ImpactAnalysis3. RefertoMOSnotes:

Ø SRDC- ZeroDataLossRecoveryAppliance(ZDLRA)DataCollection(DocID2154189.1)Ø ZeroDataLossRecoveryApplianceSystemActivityScript(DocID2275176.1)Ø ZDLRADetailedTroubleshootingMethodology(DocID2408256.1)Ø HowtocreateaTechnicalServiceRequest(SR)inMyOracleSupport(DocID1321379.1)Ø HowtoCreateServiceRequestsforSoftwareIssuesUsingaHardwareSupportIdentifier(DocID1439980.1)Ø Whotocontacttodiscussdetails,correctionsorchangestoaSupportContract(DocID1250376.1)

4. RemembertoescalateanSRwhichisnotprogressingtoyoursatisfaction.Escalationbringsmanagementattentiontotheissue.YoucanescalateanSRofanyseverity

63


Escalations:BringingManagementAttentiontoyourServiceRequest• AskingtoincreasetheseverityofyourservicerequestisNOT anescalation,eventhoughcustomers

followthesameprocess

• AnescalationisbringingOracleSupportManagement’sattentiontoyourservicerequest– Proactivereasonstoescalateanissue:

• CommunicatebusinessissuestomanagerswithinOracleSupport– Reactivereasonstoescalateanissue

• Encounteringcriticalroadblocks• Dissatisfiedwithresolutionorresponse

• DescribeImpactofProblem:Projectdeadlines?,LostRevenue?,Governmentreporting?IncreaseinRecoverabilityandDataLossPotential

64


HowtoEscalateorChangeSeverity1. Callthe24x7SupportHotline

TollFree:0800.891.5899http://www.oracle.com/support/contact.html

2. Chooseoption#1foranexistingservicerequest

3. EnteralldigitsoftheSRnumberfollowedby#

4. ChoosetheoptiontoESCALATE theSRDoNOT choosetheoptiontospeaktotheowningengineer,youwanttospeaktoanEscalationManager (formerlycalledDutyManager)inordertoescalateorchangetheseverityoftheSR.AlwaysrequestacallbackfromtheEscalationManagerwhenescalatinganSR!

• EngineeredSystemsHardwareReplacements:ChoosetheoptionforFieldDeliveryDispatchtoschedule,changeorcheckthestatusofaFieldEngineervisitforhardwarereplacement

65


WhenCallingforEngineeredSys.Support…Thefollowingoptionswilldirectyourcalltotherelevantteam:• Press"1" forExistingServiceRequests– EnterServiceRequest#.Iflookupissuccessful:

– Press“1”tospeaktotheengineerworkingyourservicerequest• IfEngineerisavailable,callwillconnect• Ifengineerisnotavailable– Press“1”toleaveVMor,– Press“2“tospeaktothenextavailableEESTengineer

– IfthereisnoFSTask,press“2”toEscalate theServiceRequest– IfthereisaFStask,• Press“2”forFieldDeliveryDispatch• Press“3”forEscalation Request

• Press“2”forNewServiceRequests– Press“1”EngineeredSystems• ForTechnicalProductIssuesPress“1”• ForNon-TechnicalIssuessuchasMyOracleSupportorSupportIdentifierIssuesPress“2”

66


SeverityLevel Business&TechnicalImpact 1st Response UpdateFrequency Resolution

Time

1 MissionCriticalBusinessImpact <1Hour(telephonepreferred)

Continual Updates24x7 Co-Owned

2 SeriousBusinessImpact CommunicationPreference

MultipleUpdates24-48hours Co-Owned

3 MinorBusinessImpact CommunicationPreference

Updates2-3BusinessDays Co-Owned

4 NoBusinessImpact CommunicationPreference

Updates3-5BusinessDays Co-Owned

Partnership

ServiceRequestSeverityLevels


PlatinumSupportforRecoveryAppliance


Agenda

69



RAStayHealthyPlan


MigrationwithZDLRA


RA Roles and Responsibilities

1

2

3

4

5

6

7


TypicalquestionsfromnewZDLRAcustomers

?

NewZDLRA

Customer

1.Who willmanagethis?

2.How willmyteamsupportit?

Thissectionprovidesinformationtohelpyouanswerthesequestions.


TheStartingPoint:ASetofSpecialistTeams

Database AdministratorsDB DB DB

Storage AdministratorsDisk DiskDiskDisk

System AdministratorsServerServer

Network AdministratorsNetwork

Security AdministratorsResponsesEventsPolicy


Option1:MultipleSpecialistTeams(statusquo1)

Database AdministratorsStorage Administrators

System Administrators

Network Administrators

Security Administrators

1- EachteamisassignedaspecificZDLRAsetofresponsibilities.


Option2:ExtendedRATeam





Primarystakeholders1

Secondarystakeholders2

1-PerfrommainZDLRAmonitoring& administrationactivities.

2-Perfromadditionaltasksrequiredbythemainstakeholders.


Option3:RAMachineAdministrator





Centralizedstakeholders1

1-ResponsibleforallZDLRAactivitiesincludingmonitoring,administrationandmaintenance.


OperationalModel1:MultipleSpecialistTeams

75

RACILegend:R =ResponsibleA =AccountableS =SupportiveC =ConsultedI =Informed

RASCIModel- MultipleSpecialistTeams

Task DBAOS

SYSADMIN NETWORK STORAGE CIO SECURITY

Day-to-dayOperation R R R A C

Testing R R R A C/IMonitoring R R R A C/IConfigurationManagement R R R A C/IPatching

StorageCells R A C/IInfiniBand C R A C/ICiscoPublicNetwork C R A C/IDbComputeNodesOS C R A C/IDatabase R A C/I

BackingUpRA R A C/IUpgradingSW R R A C/IReplacingHW R R A C/IMeteringandCharging






OperationalModel2:ExtendedRATeam(ERMA)

76


RASCIModel– ExtendedRATeam

Task DBAOS


Day-to-dayOperation R S S A C

Testing R S S A C/IMonitoring R A C/IConfigurationManagement R A C/IPatching

StorageCells R A C/IInfiniBand R S A C/ICiscoPublicNetwork R S A C/IDbComputeNodesOS R S A C/IDatabase R A C/I

BackingUpRA R A C/IUpgradingSW R S S A C/IReplacingHW R S S A C/IMeteringandCharging






OperationalModel3:RAMachineAdministrator(RAMA)

77

RASCIModel- RAMACHINE ADMINISTRATOR

Task DBAOS


Day-to-dayOperation R A C

Testing R A C/IMonitoring R A C/IConfigurationManagement R C A C/IPatching

StorageCells R A C/IInfiniBand R C/I A C/ICiscoPublicNetwork R C/I A C/IDbComputeNodesOS R A C/IDatabase R A C/I

BackingUpRA R A C/IUpgradingSW R A C/IReplacingHW R A C/IMeteringandCharging







SummaryofStaffingModels

Allthreemodelswillwork.

Youdecidewhatisbestinyourorganization.


OpenWorld2018PresentationsforRecoveryAppliance

• ZeroDataLossRecoveryAppliance:LeveragingIntegrationwithOracleCloud• ZeroDataLossRecoveryAppliance:Insider’sGuidetoArchitectureandBestPractices• OracleRecoveryManager:LatestGenerationFeaturesforOn-PremisesandtheCloud

ZDLRADocSet:“OracleZeroDataLossRecoveryAppliance”withinhttps://docs.oracle.com/en/engineered-systems/

Documents

Zero Data LossRecoveryAppliance - Oracle · •Use Recommended RA Software to avoid known critical issues –Number 1 method to avoid problems: Upgrade to recommended software release