Click here to load reader

Lessons Learned in Utility Management - Glag-Consultglag-consult.com/pic/peu99.pdfsyscopy xxx x sysdisc x syserr xx sysin x xxx xx xx xxx xx xx x sysmap x sysprint x xxx xx xx xxx

  • View
    240

  • Download
    5

Embed Size (px)

Text of Lessons Learned in Utility Management - Glag-Consultglag-consult.com/pic/peu99.pdfsyscopy xxx x...

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Jrgen GlagSOFTWARE ENGINEERING GmbH

    Dsseldorf, Germany

    [email protected]@seg.de

    Copyright Jrgen Glag, 1999 foil 01/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Low consumption of CPU and elapsed time

    Compatibility with application systems

    Minimal utility abends

    Copyright Jrgen Glag, 1999 foil 02/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Low consumption of CPU and elapsed time

    Why?Machine resources primarily required forapplication systems

    How?Utilities performed only on an as needed basis

    Fast utilities

    Copyright Jrgen Glag, 1999 foil 03/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Compatibility with application systems

    Why?Reduced batch window

    Increased need for 24 x 7 operation

    How?Reduce locking incurred by utilities

    But: Possible consequences for recoveryCopyright Jrgen Glag, 1999 foil 04/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Minimal utility abends

    Why?No restart required with negative impacton applications = less resource consumption

    Higher availability of DB2 objects = applicationsystems

    How?Test, Test, TestCorrect integration in job schedulersControlling utilities during run-time

    Copyright Jrgen Glag, 1999 foil 05/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Scope of Utilities

    Online Utilities = Utilities under DSNUTILB

    CATMAINT CHECK DATA CHECK INDEXCOPY DIAGNOSE LOADMERGECOPY MODIFY QUIESCERECOVER TABLESPACE RECOVER INDEX REORGREPAIR REPORT RUNSTATSSTOSPACE

    Return Codes:

    0 normal completion4 completion with warning messages8 failure12 authorization error

    Copyright Jrgen Glag, 1999 foil 06/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Reasons for Failure

    DB2 terminates utility step Action: New job, same utility identifier

    DB2 does not execute a specific utility function,but prior functions

    Action: New step to execute the function

    Utility function is placed in stopped state Action:

    Restart job step, same utility identifieror-TERM UTIL(utilid) and resubmit

    Copyright Jrgen Glag, 1999 foil 07/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Restart Options

    Restart of Utility

    PHASE restart: from the beginning of the phase CURRENT restart: from the last checkpoint

    Restart Step (= DSNUTILB)

    Change //ddname DD DISP=NEW,...To //ddname DD DISP={OLD/MOD},...

    Change // DSN=dsname(+1)to // DSN=dsname(0)

    Copyright Jrgen Glag, 1999 foil 08/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    DD Names and Utilities

    Copyright Jrgen Glag, 1999 foil 09/39

    CATMAINT

    CHECK

    DATA

    CHECK

    IX

    COPY

    DIAGNOSE

    LOAD

    MERGECOPY

    MODIFY

    QUIESCE

    RECOV

    TS

    RECOV

    IX

    REORG

    REPAIR

    REPORT

    RUNSTATS

    STOSPACE

    DATAWKnn XDSSPRINT XSORTLIBSORTOUT X X XSORTWKnn X X X X XSYSCOPY X X X XSYSDISC XSYSERR X XSYSIN X X X X X X X X X X X X X X X XSYSMAP XSYSPRINT X X X X X X X X X X X X X X X XSYSREC X XSYSUT1 X X X X X XUTPRINT X X X X X

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Restart due to SB37 on Output Datasets Utilities restartable at Commit points

    Copyright Jrgen Glag, 1999 foil 10/39

    Utility Phase Output Dataset

    COPY Copy SYSCOPYCOPYDDNRECOVERYDDN

    MERGECOPY Mergecopy SYSUT1SYSCOPYRECOVERYDDN

    LOAD Reload SYSUT1SYSERRSYSMAP

    REORG Unload(SORTDATA notspecified)

    SYSREC

    REORG Reload SYSUT1SYSERRSYSMAP

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Restart due to SB37 on Output Datasets (contd)

    Copy output DS to temporary DS same DCB no reblock (DFDSS ADRDSSU or DFSORT

    ICEGENER) Delete and define output DS with more space

    same VOLSER, DS-Name, DCB Copy temporary DS into output DS

    no reblock (see above) Restart utility at last commit point

    Never use IEBGENER or ISPF 3.3 for copyoperation

    Copyright Jrgen Glag, 1999 foil 11/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Utility abends

    DB2 abends MVS abends Return Codes

    Organizational errors / inefficiencies

    Example: Recover TOCOPY impossible

    Copyright Jrgen Glag, 1999 foil 12/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Utility DB2 abends and reactions

    Copyright Jrgen Glag, 1999 foil 13/39

    Utility PhaseMVS CC

    MVS RC

    DB2 Reason Code Action

    CHECK DATA, CHECK INDEX Any Any Any Any -TERM UTIL

    COPY Any Any Any Any -TERM UTIL

    S04ETry again, then-TERM UTIL

    S04E 00C9008E

    (-904,-911,-913,-923: Timeout during lock request)Try again, then -TERM

    S04E 00C90080

    (Object started for RO access, RW required)Try again, then -TERM

    S04E 00C90082(Timeout, Resource in use)Try again, then -TERM

    REORG IX S04EDeadlockTry again, then -TERM

    REORG IX Build S04EDeadlock in Build PhaseTry again, then Recover Index

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Utility MVS abends and reactions

    Copyright Jrgen Glag, 1999 foil 14/39

    Utility PhaseMVS CC

    MVS RC

    DB2 Reason Code Action

    COPY SA13Error during OPEN SYSCOPY-TERM

    SB37 More Space, then RESTART

    S222Cancel by operating,-TERM

    COPY S04E E40071I/O error on workfile,-TERM

    REORG IXBuild /Sort S04E Recover Index

    REORG IX Unload S04E -TERM

    REORG TSBuild /Sort S04E Recover Index

    REORG TS Reload S04E Recover TablespaceREORG TS Unload S04E -TERM

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Utility Return codes and reactions

    Copyright Jrgen Glag, 1999 foil 15/39

    UtilityMVS CC

    Msg IDRC Action

    COPY 8 DSNU054ITS not foundSet RC to 4

    8 DSNU180IDifferent util on TSTry again

    REORG IX 8 DSNU055IIX not foundSet RC to 4

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for COPY

    FIC (Full Image Copy) after LOAD LOG NO REORG LOG NO RECOVER TORBA orTOCOPY > 10% changed pages

    (cheaper than Incremental Copy)

    RECOVER requires all relevant Archive Logs atonce

    try to recover without archive logs indicator for frequency of copies

    Copyright Jrgen Glag, 1999 foil 16/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for COPY (contd)

    Always take dual copies (media failure!)

    For speeding up recovery have at least 2successive full copies with incremental copiesin between

    Catalog and Directory should be copied alwaystogether

    DFSMS Concurrent Copy ICTYPE = F, STYPE = C No Incremental Copy possible

    Copyright Jrgen Glag, 1999 foil 17/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for Archive Logs

    No Archive logs from different subsystems (of adata sharing group) on same tape

    Recovery can be inhibited!!!

    Recovery from Archive logs on tape mayrequire more tape units than you have

    Solution: Frequent Image Copies and recovery points(QUIESCE)

    Copyright Jrgen Glag, 1999 foil 18/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for Active Logs

    Dual Active logs separate controllers separate channels separate volumes

    One log fails Recover from the other log Update BSDS

    Copyright Jrgen Glag, 1999 foil 19/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for Active Logs (contd)

    Both logs fail No recovery possible Cold start DB2 Full Image Copy immediately

    Data Sharing: Changes of one object may be written to all log

    datasets on all subsystems

    Copyright Jrgen Glag, 1999 foil 20/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for BSDS

    Dual copies

    One BSDS fails Recover from the other

    Both BSDS fail Recover from most recent Archive Log

    Active -> Archive Log: BSDS always included

    Copyright Jrgen Glag, 1999 foil 21/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for RECOVER

    RECOVER TOCOPY IC with SHRLEVEL REFERENCE

    IC with SHRLEVEL CHANGE

    RECOVER TORBA IC with SHRLEVEL CHANGE + QUIESCE

    Copyright Jrgen Glag, 1999 foil 22/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for RECOVER (contd)

    RECOVER dropped object Recreate like last IC

    Be careful with DDL changes Recover utility cannot be used (no OBID translation) Instead: DSN1COPY

    Logs cannot be applied

    Solutions: CREATE TABLE ... WITH RESTRICT ON DROP Frequent Copies

    Copyright Jrgen Glag, 1999 foil 23/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Tips for RECOVER (contd)

    Catalog and Directory Recover to same point 2 separate utility control statements required

    When data sharing is disabled: immediate backup of whole system otherwise: no recovery possible

    Copyright Jrgen Glag, 1999 foil 24/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    General Recommendations

    Primary objective:

    Availability of application systems

    Copyright Jrgen Glag, 1999 foil 25/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    General Recommendations

    COPY SHRLEVEL CHANGE for better availability of

    applications,but: QUIESCE before and after

    DSNUM where applicable

    DFSMS Concurrent Copy for better availability ofapplications

    LOAD LOG NO with inline COPY (COPYDDN)

    Copyright Jrgen Glag, 1999 foil 26/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    General Recommendations

    REORG LOG NO and inline COPY (COPYDDN)

    SHRLEVEL CHANGE for better availability ofapplications,but: QUIESCE before and after

    Copyright Jrgen Glag, 1999 foil 27/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy Types

    Execution of utilities in a time block on week-ends

    Distribution of utility execution, but still nocommon time intervals for applications andutilities

    Utilities and applications share a common timeframe

    Copyright Jrgen Glag, 1999 foil 28/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #1:Week-ends are reserved for Utilities

    Execution of utilities (COPY, REORG) in a timeblock on week-ends

    Full Image Copy of every tablespace Reorg of every tablespace

    Incremental Image Copy on a daily basis

    Copyright Jrgen Glag, 1999 foil 29/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #1:Week-ends are reserved for Utilities

    Scenarios Small operational application systems Relatively low update rate, therefore no need for

    intensive Copy and Reorg Read-only application systems Packaged application systems (SAP R/3, Peoplesoft)

    with a low number of users Limits

    Rising number of transactions and updates Increased number of users, esp. with packaged

    systems

    Copyright Jrgen Glag, 1999 foil 30/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #2:Everyday is Utility day - with fixed schedules

    COPY, REORG executed daily in an own timeframe, i.e. after online and batch

    Fixed schedule for COPY of tablespaces Monday: COPY TS1, ..., TS1000

    Tuesday: COPY TS1001, ..., TS2000...

    Fixed schedule for REORG of tablespacesand/or indexes

    schedule not based on update rate or growthCopyright Jrgen Glag, 1999 foil 31/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #2: Everyday is Utility day - with fixed schedules

    Scenarios Small operational application systems Low / medium update rate Batch window for utilities still sufficient

    Limits Rising number of DB2 objects and corresponding

    updates Clusterratio can differ significantly, therefore

    performance degradation on particular objects Highly different update rates imply different time

    required for recovering particular tablespacesCopyright Jrgen Glag, 1999 foil 32/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #3:Deployment of Fast Utilities

    BMC, CA, CDB, ...

    Utility execution comparable to strategy #2, i.e.fixed schedules

    Insufficient time to execute standard IBMutilities COPY, REORG in a dedicated batchwindow

    Copyright Jrgen Glag, 1999 foil 33/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #3:Deployment of Fast Utilities

    Benefits: Formerly up to 8-10 times faster than IBM utils

    => Sufficient batch window for utils

    Fast Utils often based on log files=> No locking conflicts=> Can be executed during application availability

    Copyright Jrgen Glag, 1999 foil 34/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #3:Deployment of Fast Utilities

    Caveats: Extra cost for license and maintenance

    => MIPS based pricing may become too expensivefor big shops

    IBM utils became faster offer better compatibility to applications due to reduced

    locking and granularity options=> Utilities fit into the available batch window=> Can be executed during application availability

    Different update rates or growth not reflectedCopyright Jrgen Glag, 1999 foil 35/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #4:Reduce to the necessary - generate only therequired utilities

    Define thresholds for utility execution grouping of DB2 objects for common thresholds reflect size of object depending on rate of changes

    Check if catalog statistics need to be refreshed Avoid RUNSTATS for checking Instead: Check Space Map pages if RUNSTATS

    necessary Generate and execute RUNSTATS to provide correct

    statisticsCopyright Jrgen Glag, 1999 foil 36/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #4:Reduce to the necessary - generate only therequired utilities

    Check if thresholds are reached Based on correct catalog statistics Change rate, growth, ... imply COPY, REORG Generate and execute only required utilities

    Copyright Jrgen Glag, 1999 foil 37/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #4:Reduce to the necessary - generate only therequired utilities

    Benefits: Applicable for standard IBM and fast utilities Space Map analysis

    => fast, suitable even for largest DB2 systems=> no locking conflicts with applications

    DB2 Catalog always represents reality=> only least necessary activities required for catalogmaintenance

    Minimal execution of COPY, REORG Tablespaces / Indexes always correspond to

    individually defined thresholdsCopyright Jrgen Glag, 1999 foil 38/39

  • Objectives forutility handling

    What can gowrong withutilities?It will ;-)

    Utility strategies -Overview

    Utility strategies -Scenarios andLimits

    Lessons Learned in Utility Management

    Strategy #4:Reduce to the necessary - generate only therequired utilities

    Caveats: ???

    Further activities Running minimal number of utilities during availability

    of application systems (care about SHRLEVEL) Deployment of fast utilities

    Copyright Jrgen Glag, 1999 foil 39/39