Starter'sguidetoREALmemorymanagementwithDB2forz/OS

JohnCampbellIBMDB2forz/OSDevelopmentSessionCode:9007Platform:DB2forz/OS

Disclaimer/Trademarks

© Copyright IBM Corporation 2017. All rights reserved.U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

THE INFORMATION CONTAINED IN THIS DOCUMENT HAS NOT BEEN SUBMITTED TO ANY FORMAL IBM TEST AND IS DISTRIBUTED AS IS. THE USE OF THIS INFORMATION OR THE IMPLEMENTATION OF ANY OF THESE TECHNIQUES IS A CUSTOMER RESPONSIBILITY AND DEPENDS ON THE CUSTOMER’S ABILITY TO EVALUATE AND INTEGRATE THEM INTO THE CUSTOMER’S OPERATIONAL ENVIRONMENT. WHILE IBM MAY HAVE REVIEWED EACH ITEM FOR ACCURACY IN ASPECIFIC SITUATION, THERE IS NO GUARANTEE THAT THE SAME OR SIMILAR RESULTS WILL BE OBTAINED ELSEWHERE. ANYONE ATTEMPTING TO ADAPT THESE TECHNIQUES TO THEIR OWN ENVIRONMENTS DO SO AT THEIR OWN RISK.

ANY PERFORMANCE DATA CONTAINED IN THIS DOCUMENT WERE DETERMINED IN VARIOUS CONTROLLED LABORATORY ENVIRONMENTS AND ARE FOR REFERENCE PURPOSES ONLY. CUSTOMERS SHOULD NOT ADAPT THESE PERFORMANCE NUMBERS TO THEIR OWN ENVIRONMENTS AS SYSTEM PERFORMANCESTANDARDS. THE RESULTS THAT MAY BE OBTAINED IN OTHER OPERATING ENVIRONMENTS MAY VARY SIGNIFICANTLY. USERS OF THIS DOCUMENT SHOULD VERIFY THE APPLICABLE DATA FOR THEIR SPECIFIC ENVIRONMENT.

TrademarksIBM, the IBM logo, ibm.com, DB2, FlashCopy, GDPS, Redbooks, System z and z/OS are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

2

Agenda• Commonproblems• Realmemorymonitoringandtuningprinciple• DUMPsettings• Toolsofchoice• Largesizerealmemorypageframes• Appendix

3

Commonproblems• Toomanyinstallationsarerunning‘oneDUMPawayfromadisaster’

• Notgoodpracticetorelyonpagingtoabsorbpeakdemands• DUMPcapture,workloadfailover,systemslowdown

• Canleadtoexcessivepagingandsevereperformance issues• CanleadtolongDUMPprocessingtimesandcausemajordisruption

• Oncepagingbegins,itispossibletohaveDUMPprocessingtake10sofsecondstominutes• Highriskofsysplex-wideslowdown/hangasp-locknegotiationisaffected

• Ultimately,cantaketheLPARout• OnceallAUXisconsumed,theLPARgoesintoawaitstate

4

Commonproblems…• WastedopportunitiesforCPUreduction

• Reluctancetousebiggerormorebufferpools• Reluctancetousebufferpoollong-termpagefix• ManyperformanceopportunitiesinDB2requirerealmemory

• e.g.RELEASE(DEALLOCATE)• Increasinglylargerrealmemorysizeswillberequiredforfutureperformanceandscalabilityimprovements

• e.g.V12indexFastTraverseBlocks

5

Realmemorymonitoringandtuningprinciple• Makesureyouhaveenoughrealmemoryto

1. Coveryourworkloadpeakdemands(peakworkingset)2. AllowaDUMPtobecapturedquicklyinmemorywithoutcausinganydisruption3. Provideenoughsparememoryforworkloadfailoverorabnormalslowdowns

• Inotherwords,designandtuneforzeropaging,evenduringDUMPcapture• Notjustaboutminimisingpagingrates• YouwanttoavoidDB2beingpagedout

• IncaseofDUMP,eachpagemayhavetobepagedin,whichmayseverelyimpactcapturetime• NoprefetchonAUXstorage,soallsynchronousI/Os

6

HowmuchrealmemoryisconfiguredontheLPAR?

7

Command D M=STOR-------------------------------------------------------------------------------

IEE174I 11.51.10 DISPLAY MREAL STORAGE STATUSONLINE-NOT RECONFIGURABLE

0M-67584MONLINE-RECONFIGURABLE

NONEPENDING OFFLINE

NONE2048M IN OFFLINE STORAGE ELEMENT(S)0M UNASSIGNED STORAGESTORAGE INCREMENT SIZE IS 256M

DUMPsettings

8

Command D DUMP,O-------------------------------------------------------------------------------

IEE857I 14.38.39 DUMP OPTION SYSABEND- ADD PARMLIB OPTIONS SDATA=(LSQA,TRT,CB,ENQ,DM,IO,ERR,SUM),

PDATA=(SA,REGS,LPA,JPA,PSW,SPLS)SYSUDUMP- ADD PARMLIB OPTIONS SDATA=(SUM), NO PDATA OPTIONSSYSMDUMP- ADD PARMLIB OPTIONS (NUC,SQA,LSQA,SWA,TRT,RGN,SUM)SDUMP- ADD OPTIONS (ALLPSA,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP,

ALLNUC,Q=YES,GRSQ,COUPLE,XESDATA,WLM,SERVERS),TYPE=(XMEME),BUFFERS=00000000K,MAXSPACE=00020480M,MSGTIME=00000 MINUTES,MAXSNDSP=015 SECONDS,AUXMGMT=ON ,DEFERTND=NO

SYSFAIL NO STRLIST OPTIONSABDUMP- TIMEENQ=0240 SECONDS

AUXMGMT specifieswhenSDUMPdatacapturesshouldstopDefaultisON(recommendedsetting)• NonewdumpsareallowedwhenAUXstorageutilisation reaches50%• CurrentdumpdatacapturestopswhenAUXstorageutilisation reaches68%• Oncethelimitisexceeded,newdumpswillnotbeprocesseduntiltheAUXstorageutilisationdropsbelow35%

MAXSPACE restrictsthevirtualstorageavailabletotheDUMPSRVaddressspace

DUMPsettings…• Makesure AUXMGMT=ON(default)

• StopcurrentaswellasnewDUMPswhenexcessivepagingonthesystem• DonotoversizeAUXotherwisesafetynetslikeAUXMGMT,ENF55willnothelpyou• MakesureMAXSPACEissetproperlyanddefensively

• Generaldefault:Planforapprox.16GBperDB2memberofthesamedatasharinggrouprunningonthesameLPAR• AllowforDUMPwildcarding• CouldbeadjustedafterbenchmarkingDUMPsizesintheproductionenvironment

• Over-sizingMAXSPACEmightresultinexcessivepagingonthesystem• Under-sizingMAXSPACEwillresultinpartialdumpsandseriouslycompromisesproblemdetermination

• Dumpsshouldbetakenveryquickly(<10secs)almostwithoutanyonenoticingi.e.,littleornodisruptiontothesubjectLPARandtotherestoftheSysplex• Considerautomationtokilldumpstakinglongerthan10secs

9

Storage-ClassMemory(SCM)onFlashExpresscards• SCMisoptionalauxiliarystorage,

inadditiontopagedatasets

• CommonmisconceptionthatSCMistheretoabsorbDUMPs• PagingtoSCMismuchfasterthanpagingtoAUX(DASD)• Butitismostunlikelythat,whenalargeDUMPistaken,itwillbefastenoughtoavoiddisruptionontheLPAR

andpossible‘sympathysickness’acrossthedatasharinggroup• SCMshouldbeseenasan‘insurancepolicy’toprotectthesystemagainstunintended pagingin

exceptionalcircumstances

10

Command D M=SCM(DETAIL)-------------------------------------------------------------------

STORAGE-CLASS MEMORY STATUS - INCREMENT DETAIL96G DEFINEDADDRESS IN USE STATUS

0G 0% ONLINE16G 0% ONLINE32G 0% ONLINE48G 0% ONLINE64G 0% ONLINE

ONLINE: 80G OFFLINE-AVAILABLE: 16G PENDING OFFLINE: 0G0% IN USE

Toolsofchoice

11

#1toolofchoice• IFCID225

• CollectedaspartofDB2StatisticsClass1• DonotjustlookatsnapshotsorStatisticsreports• CreategraphsfromDB2starttoDB2shutdowntostudytheevolutionarytrend

• OptionstopullIFCID225data• PerformancedatabasewhereStatisticsdataiskeptatone-minuteintervals• SampleREXXprogramMEMU2topullthedataviaIFIandcreatea.csvfile

• AvailablefordownloadonIBMMydeveloperWorks – NewversionsupportingDB212!https://www.ibm.com/developerworks/community/files/app?lang=en#/file/deac40b6-58de-4816-82f4-2d8dbd75416aMakesureMEMU2jobrunswithhighdispatchingpriorityotherwisesystemmaystallonlatch

• OMPESpreadsheetInputDataGeneratortopostprocessSMFdataandcreate.csvfiles• APARPM73732andOMPEv510PTFUK90267/OMPEv511PTFUK90268• SampleJCLprovidedinAppendix

12

Memorymap• 31-bitand64-bitprivatememoryforeachDB2systemaddressspace

• NosharingacrossDB2systemaddressspaces• LargestcontributorsinceDB210:DBM164-bitprivate

• Bufferpools,dynamicstatementcache,skeletonpool,DBDpool,compressiondictionaries• LOBandXMLmemoryusage

• 64-bitsharedmemory(HVSHARE)• AllowssharingacrossDB2systemaddressspacesandutilities(tokenrequiredtoconnect)

• Noneedtouseexpensivecrossmemorymoves• Reducedcomplexity,addressablestorageisalwaysavailable

• Useof64-bitsharedmemorynowdominatesinDB2• Thread-relatedstorage(systemandnon-systemagents)• Sharedstack(saveareas,workingstorage)

• 31-bitand64-bitcommonmemory– CSA/ECSAandHCSA(HVCOMMON)• SmalluseofCSA/ECSA(butifmanyDB2subsystemsperLPARthenaggregaterequirementmaybesignificant)• Limiteduseof64-bitcommoninDB2V10,butlogbuffersaremovedthereinV11CM

13

Graph#1– Realmemorycounters

14

SeeappendixforIFCID225fieldnamesandmappingreferencetableforMEMU2andOMPEperformancedatabase

DBM1REALinusefor31and64-bitpriv

DBM1REALinusefor64-bitpriv

DBM1REALinusefor64-bitprivw/oBP

DBM1REALinusefor31-bitpriv =DBM1REALinusefor31and64-bitpriv – DBM1REALinusefor64-bitpriv

DBM1REALinuseforBP=DBM1REALinusefor64-bitpriv – DBM1REALinusefor64-bitpriv w/oBP

DISTREALinusefor64-bitpriv

DISTREALinusefor31and64-bitpriv

DISTREALinusefor31-bitpriv =DISTREALinusefor31and64-bitpriv – DISTREALinusefor64-bitpriv

REALinusefor64-bitshared

REALinusefor64-bitsharedstack

REALinusefor64-bitcommon

0

100

200

300

400

500

600

700

800

0

2000

4000

6000

8000

10000

12000

09:23:01

10:03:02

10:43:02

11:23:02

12:03:02

12:43:03

13:23:03

14:03:03

14:43:04

15:23:04

16:03:04

16:43:04

17:23:05

18:03:05

18:43:05

19:23:05

20:03:06

20:43:06

21:23:06

22:03:06

22:43:07

23:23:07

00:03:07

00:43:07

01:23:08

02:03:08

02:43:08

03:23:08

04:03:09

04:43:09

05:23:09

06:03:09

06:43:09

07:23:10

08:03:10

08:43:10

(MB)TotalREALstorageinuse

ASIDDBM1REALinusefor31-bitpriv(MB) ASIDDBM1REALinusefor64-bitprivw/oBP(MB)

ASIDDBM1REALinuseforBP(MB) ASIDDISTREALinusefor31-bitpriv(MB)

ASIDDISTREALinusefor64-bitpriv(MB) REALinusefor64-bitshared(MB)

REALinusefor64-bitsharedstack(MB) REALinusefor64-bitcommon(MB)

Current#ofactivethreads

15

Graph#2– AUXstoragecounters

16

SeeappendixforIFCID225fieldnamesandmappingreferencetableforMEMU2andOMPEperformancedatabase

WARNING!(1) TherecouldbedoubleaccountingwithREAL(ifaframewaspagedoutandpagedbackin,itmayshowupbothinREALandAUX)(2) DoesnotincludeframesthatareinSCM

DBM1AUXinusefor31and64-bitpriv(1)(2)

DBM1AUXinusefor64-bitpriv(1)(2)

DBM1AUXinusefor64-bitprivw/oBP

DBM1AUXinusefor31-bitpriv =DBM1AUXinusefor31and64-bitpriv – DBM1AUXinusefor64-bitpriv

DBM1AUXinuseforBP=DBM1AUXinusefor64-bitpriv – DBM1AUXinusefor64-bitpriv w/oBP

DISTAUXinusefor64-bitpriv(1)(2)

DISTAUXinusefor31and64-bitpriv(1)(2)

DISTAUXinusefor31-bitpriv =DISTAUXinusefor31and64-bitpriv – DISTAUXinusefor64-bitpriv

AUXinusefor64-bitshared

AUXinusefor64-bitsharedstack

AUXinusefor64-bitcommon

0

0

0

0

0

1

1

1

1

1

1

09:23:01

09:58:02

10:33:02

11:08:02

11:43:02

12:18:02

12:53:03

13:28:03

14:03:03

14:38:03

15:13:04

15:48:04

16:23:04

16:58:04

17:33:05

18:08:05

18:43:05

19:18:05

19:53:06

20:28:06

21:03:06

21:38:06

22:13:06

22:48:07

23:23:07

23:58:07

00:33:07

01:08:08

01:43:08

02:18:08

02:53:08

03:28:08

04:03:09

04:38:09

05:13:09

05:48:09

06:23:09

06:58:10

07:33:10

08:08:10

08:43:10

09:18:10

(MB) TotalAUXstorageinuse

ASIDDBM1AUXinusefor31-bitpriv(MB) ASIDDBM1AUXinusefor64-bitprivw/oBP(MB)

ASIDDBM1AUXinuseforBP(MB) ASIDDISTAUXinusefor31-bitpriv(MB)

ASIDDISTAUXinusefor64-bitpriv(MB) AUXinusefor64-bitshared(MB)

AUXinusefor64-bitsharedstack(MB) AUXinusefor64-bitcommon(MB)

TotalAUXinuse-DB2

17

0

20

40

60

80

100

120

2014-04-28-00.02

.00.00

0101

2014-04-28-12.02

.00.00

0126

2014-04-29-00.02

.00.00

0646

2014-04-29-12.02

.00.00

0163

2014-04-30-00.02

.00.00

0111

2014-04-30-12.02

.00.00

0118

2014-05-01-00.02

.00.00

0903

2014-05-01-12.02

.00.00

0075

2014-05-02-00.02

.00.00

1633

2014-05-02-12.02

.00.00

0136

2014-05-03-00.02

.00.00

0078

2014-05-03-12.02

.00.00

0128

2014-05-04-00.02

.00.00

0110

2014-05-04-12.02

.00.00

0105

2014-05-05-00.03

.00.00

0125

2014-05-05-12.03

.00.00

0101

2014-05-06-00.03

.00.00

0099

2014-05-06-12.03

.00.00

0105

2014-05-07-00.03

.00.00

0156

2014-05-07-12.03

.00.00

0099

2014-05-08-00.03

.00.00

0074

2014-05-08-12.03

.00.00

0097

2014-05-09-00.03

.00.00

0077

2014-05-09-12.03

.00.00

0092

2014-05-10-00.03

.00.00

0114

2014-05-10-12.03

.00.00

0153

2014-05-11-00.03

.00.00

0238

2014-05-11-12.03

.00.00

0074

2014-05-12-00.04

.00.00

0107

2014-05-12-12.04

.00.00

0140

2014-05-13-00.04

.00.00

0108

2014-05-13-12.04

.00.00

0119

2014-05-14-00.04

.00.00

1056

2014-05-14-12.04

.00.00

0086

2014-05-15-00.04

.00.00

0145

2014-05-15-12.04

.00.00

0227

2014-05-16-00.04

.00.00

0097

2014-05-16-12.04

.00.00

0159

2014-05-17-00.04

.00.00

0153

2014-05-17-12.04

.00.00

0112

2014-05-18-00.04

.00.00

0212

2014-05-18-12.04

.00.00

0101

(MB)TotalAUXstorageinuse

ASIDDBM1AUXinusefor31-bitpriv(MB) ASIDDBM1AUXinusefor64-bitprivw/oBP(MB)

ASIDDBM1AUXinuseforBP(MB) ASIDDISTAUXinusefor31-bitpriv(MB)

ASIDDISTAUXinusefor64-bitpriv(MB) AUXinusefor64-bitshared(MB)

AUXinusefor64-bitsharedstack(MB) AUXinusefor64-bitcommon(MB)

TotalAUXinuse-DB2

Inthisparticularcase,allthebufferpoolsweredefinedasPGFIX(YES),soitistherestofDB2thatispagedout(threadcontrolblocks,bufferpoolcontrolblocks,etc.)

DB2 working set is being pushed out to AUX

18

Graph#3– QW0225_REALAVAIL(S)

19

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

(blank

)20

15-10-3101:34

2015-10-3104:49

2015-10-3108:07

2015-10-3111:23

2015-10-3114:39

2015-10-3117:54

2015-10-3121:09

2015-11-0100:25

2015-11-0103:42

2015-11-0106:56

2015-11-0110:14

2015-11-0113:30

2015-11-0116:46

2015-11-0120:03

2015-11-0123:19

2015-11-0202:34

2015-11-0205:52

2015-11-0209:10

2015-11-0212:26

2015-11-0215:42

2015-11-0218:57

2015-11-0222:13

2015-11-0301:32

2015-11-0304:47

2015-11-0308:03

2015-11-0311:17

2015-11-0314:32

2015-11-0317:48

2015-11-0321:03

2015-11-0400:20

2015-11-0403:34

2015-11-0406:49

2015-11-0410:05

2015-11-0413:20

2015-11-0416:36

2015-11-0419:50

2015-11-0423:05

2015-11-0502:22

2015-11-0505:38

2015-11-0508:55

2015-11-0512:10

2015-11-0515:26

2015-11-0518:44

2015-11-0522:00

2015-11-0601:16

2015-11-0604:30

2015-11-0607:46

2015-11-0611:02

2015-11-0614:20

2015-11-0617:38

2015-11-0620:54

(GB)REALmemoryavailableonLPARSYSP

Graph#3– QW0225_REALAVAIL(S)…

20

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

(blank

)20

15-10-3101:34

2015-10-3104:49

2015-10-3108:07

2015-10-3111:23

2015-10-3114:39

2015-10-3117:54

2015-10-3121:09

2015-11-0100:25

2015-11-0103:42

2015-11-0106:56

2015-11-0110:14

2015-11-0113:30

2015-11-0116:46

2015-11-0120:03

2015-11-0123:19

2015-11-0202:34

2015-11-0205:52

2015-11-0209:10

2015-11-0212:26

2015-11-0215:42

2015-11-0218:57

2015-11-0222:13

2015-11-0301:32

2015-11-0304:47

2015-11-0308:03

2015-11-0311:17

2015-11-0314:32

2015-11-0317:48

2015-11-0321:03

2015-11-0400:20

2015-11-0403:34

2015-11-0406:49

2015-11-0410:05

2015-11-0413:20

2015-11-0416:36

2015-11-0419:50

2015-11-0423:05

2015-11-0502:22

2015-11-0505:38

2015-11-0508:55

2015-11-0512:10

2015-11-0515:26

2015-11-0518:44

2015-11-0522:00

2015-11-0601:16

2015-11-0604:30

2015-11-0607:46

2015-11-0611:02

2015-11-0614:20

2015-11-0617:38

2015-11-0620:54

(GB)REALmemoryavailableonLPARSYSP

MAXSPACE

GREENcondition

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

45.00

50.00

(blank

)20

15-10-3101:34

2015-10-3104:49

2015-10-3108:07

2015-10-3111:23

2015-10-3114:39

2015-10-3117:54

2015-10-3121:09

2015-11-0100:25

2015-11-0103:42

2015-11-0106:56

2015-11-0110:14

2015-11-0113:30

2015-11-0116:46

2015-11-0120:03

2015-11-0123:19

2015-11-0202:34

2015-11-0205:52

2015-11-0209:10

2015-11-0212:26

2015-11-0215:42

2015-11-0218:57

2015-11-0222:13

2015-11-0301:32

2015-11-0304:47

2015-11-0308:03

2015-11-0311:17

2015-11-0314:32

2015-11-0317:48

2015-11-0321:03

2015-11-0400:20

2015-11-0403:34

2015-11-0406:49

2015-11-0410:05

2015-11-0413:20

2015-11-0416:36

2015-11-0419:50

2015-11-0423:05

2015-11-0502:22

2015-11-0505:38

2015-11-0508:55

2015-11-0512:10

2015-11-0515:26

2015-11-0518:44

2015-11-0522:00

2015-11-0601:16

2015-11-0604:30

2015-11-0607:46

2015-11-0611:02

2015-11-0614:20

2015-11-0617:38

2015-11-0620:54

(GB)REALmemoryavailableonLPARSYSR

Graph#3– QW0225_REALAVAIL(S)…

21

MAXSPACE

AMBERcondition(longdropsand/ordropstoverylowvalues)

Nextsteps• Findthetimeintervalswhenproblematiclong,deepdropsorpage-outoccurred• Usethisinformationasastartingpointtofindtherootcauseofthesedrops

• Whattypeofactivitydrivessuddenincreaseinmemoryconsumption?• Usualsuspects

• DFSORTe.g.DB2utilities• DUMPcapture• RecyclingofCICSregions• Bigbatchjob• Shiftinworkload

22

#2toolofchoice• RMF

• MonitorIII– STORF(Option3.7)Storageusagebyframes

23

Realmemoryconsumptionper

jobnameà privatememory

#2toolofchoice• RMF

• MonitorIII– STORM(Option3.7A)64-BitSharedandCommon

24

64-bitsharedand64-bitcommonmemoryusage

Donotforgettoincludethemintheoverallrealmemory

consumption!

ControluseofrealmemorybyDFSORT• DFSORTdefaultsettings

• Recommendationstocontroluseofrealmemory byDFSORT• SetEXPOLD=0topreventDFSORTfromtakingoldframesfromotherworkloads• SetEXPMAXdowntolimitmaximumDFSORTusage• SetEXPRES=n{reserveenoughforMAXSPACE}

25

SampleJCLtoextractDFSORTinstallationdefaults

Beforez/OS2.1 Afterz/OS2.1EXPMAX=MAXßMakemaximumuseofmemoryEXPOLD=MAXß AllowpagingofoldframesEXPRES=0ß Noreservefornewwork

EXPMAX=MAXEXPOLD=50%EXPRES=10%

Realmemoryallocatedinfixedincrementsizes TUNE=STOR(newsetting)Gradualallocationofrealmemoryinsmallerincrements%forEXPMAX,EXPRES,EXPOLD=%ofcurrentresourcesinsteadof%ofconfiguredmemory

//SHOWDEF EXEC PGM=ICETOOL//TOOLMSG DD SYSOUT=*//DFSMSG DD SYSOUT=*//LIST1 DD SYSOUT=*//TOOLIN DD *DEFAULTS LIST(LIST1)

/*

ControluseofrealmemorybyDB2• Unlessyouhaveamplerealmemoryheadroomandcantoleratesomememorygrowth,recommend

runningwithREALSTORAGE_MANAGEMENT=AUTO(default)• WithRSM=AUTO,DB2willregularlydiscardunusedrealmemoryframes

• RSM=AUTOwithnopaging(AUTO-OFF)à ‘SoftDiscard’atThreadDeallocationorafter120commits• RSM=AUTOwithpaging(AUTO-ON)à ‘SoftDiscard’atThreadDeallocationorafter30commits– STACKalsoDISCARDED

• Prosofthe‘SoftDiscard’• ReducedREALmemoryuse• ReducedexposuretoSPINlockcontention

• Cons:• SmallCPUoverheadinMSTR/DISTSRBtime(whichcanbeminimised withgoodthreadreuse)• 64-bitsharedandcommonrealmemorycountersarenotaccurateuntilpagingoccurs

• During‘SoftDiscard’,DB2usesDISCARDwithKEEPREAL=YES• Thememoryisonly‘virtuallyfreed’• RSMflagsthepageasfreedorunused,buttheframeisstillchargedagainstDB2

26

ControluseofrealmemorybyDB2…

• Useautomationtotrap‘contractionmode’messagesà veryseverecondition• Whensignificantpagingisdetected(ENF55signal)orREALSTORAGE_MAXisreached,DB2 enters‘contraction

mode’(‘HardDiscard’)• AppliestoREALSTORAGE_MANAGEMENT=AUTO|ON|OFF• During‘HardDiscard’,DB2usesDISCARDwithKEEPREAL=NO• KEEPREAL(NO)tellsRSMtofreeandreclaimthepageimmediately

27

DSNV516I -TZE1 DSNVMON - BEGINNING STORAGE CONTRACTION MODEDSNV517I -TZE1 DSNVMON - ENDING STORAGE CONTRACTION MODE

ControluseofrealmemorybyDB2…• RefrainfromusingPGFIX(YES)ifyoudon’thaveenoughrealmemoryheadroom

• IfbufferpoolsaredefinedwithPGFIX(YES),considerspecifyingz/OSWLMSTORAGECRITICALfortheDB2systemaddressspaces• Long-termstorageprotectionfortheDB2addressspaces• Objective:HelpsafeguardtherestofDB2

• Trytokeepthreadcontrolblocks,EDMandotherneededpartsofDB2inrealmemory• MighthelppreventperformanceproblemsastheOnlinedaystartsandDB2hastoberapidlypagedbackin• Butnot100%guaranteethatDB2willnotbepagedout!

• IfyouhavemultipleDB2subsystemswithdifferingimportancerunningontheLPAR,restricttheuseofPGFIX(YES)andWLMSTORAGECRITICALtothemostimportantone

28

ControluseofrealmemorybyDB2…• ReviewyourDB2memorybudget

• Settingsthatcanhaveadirectimpactonrealmemoryusageiftheyneedtobebacked• Bufferpoolsizes(VPSIZE)• Globaldynamicstatementcache(ZPARMEDMSTMTC)• Localdynamicstatementcache(ZPARMMAXKEEPD)• DBDpoolsize(ZPARMEDMDBDC)• Skeletonpool(ZPARMEDM_SKELETON_POOL)• RIDpool(ZPARMMAXRBLK)• Sortpool(ZPARMSRTPOOL– perthreadsetting)• Logoutputlogbuffer(ZPARMOUTBUFF)• Hashingentriesindynamicstatementcache(ZPARMCACHE_DEP_TRACK_STOR_LIM)à Default2GB,SAPrecommends10GB• LOBvaluestorage(ZPARMLOBVALS)à Default=2GB• XMLvaluestorage(ZPARMXMLVALS)à Default=10GB

• IndirectlyZPARMsCTHREAD,MAXDBAT,DSMAXwillalsoaffecttheoverallbudget

29

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

18.00

(blank

)20

15-10-3101:33

2015-10-3104:48

2015-10-3108:06

2015-10-3111:22

2015-10-3114:38

2015-10-3117:53

2015-10-3121:08

2015-11-0100:24

2015-11-0103:41

2015-11-0106:55

2015-11-0110:13

2015-11-0113:29

2015-11-0116:45

2015-11-0120:02

2015-11-0123:18

2015-11-0202:33

2015-11-0205:51

2015-11-0209:09

2015-11-0212:25

2015-11-0215:41

2015-11-0218:56

2015-11-0222:12

2015-11-0301:31

2015-11-0304:46

2015-11-0308:02

2015-11-0311:16

2015-11-0314:31

2015-11-0317:47

2015-11-0321:02

2015-11-0400:19

2015-11-0403:33

2015-11-0406:48

2015-11-0410:04

2015-11-0413:19

2015-11-0416:35

2015-11-0419:49

2015-11-0423:04

2015-11-0502:21

2015-11-0505:37

2015-11-0508:54

2015-11-0512:09

2015-11-0515:25

2015-11-0518:43

2015-11-0521:59

2015-11-0601:15

2015-11-0604:29

2015-11-0607:45

2015-11-0611:01

2015-11-0614:19

2015-11-0617:37

2015-11-0620:53

(GB)REALmemoryavailableonLPARSYSQ

QW0225_REALAVAIL(S)

30

MAXSPACE

REDcondition

Nextsteps• Buymorememory!

31

Largesizerealmemorypageframes• Benefitoflargesize(1Mor2G)realmemorypageframes

• TheTranslationLookasideBuffer(TLB)isacacheusedtospeeduptheconversionofvirtualaddressesintorealaddresses

• Withtheintroductionof64-bitrealandvirtualaddressing,theTLBcoveragehasdramaticallyshrunk,leadingtoperformancedegradation

• LargesizepageframeshelpincreaseTLBcoveragewithouthavingtoenlargetheTLBsize• Result:BetterperformancebydecreasingthenumberofTLBmisses

• Commonproblem• LFAREAisgrosslyover-configured,whichmightresultinashortageof4Ksizeframesandleadtoexpensive

breakdownof1Msizelargeframes,expensivepagemovementfor4Kpagefixes,and/orprematurepaging

32

Largesizerealmemorypageframes…• LFAREA– 1M/2Glargeframearea

• Fixed1M/2Gpageframes• DefinedinIEASYSxxparmlibmember• ‘Old’syntax(stillsupported)

• LFAREA=(xM|xG|xT|x%)• Pctformula:(x%ofonlinememoryavailableatIPL)– 2G• MaxLFAREAis(80%*onlinerealmemory availableatIPL)– 2G

• Newsyntax• LFAREA=(1M=(a[,b])|1M=(a%[,b%])|2G=(a[,b])|2G=(a%[,b%])• Pctformula:x%of(onlinememory availableatIPL– 4G)• MaxLFAREAis80%of(onlinememory availableatIPLtime– 4G)

• OnlychangeablebyIPL• IftheLFAREAisovercommitted,DB2willuse4Kand/or1Msizepageframes

33

Largesizerealmemorypageframes…• Quadarea

• 12.5%ofonlinememory atIPLtime

• PLArea– Pageable1Mlargeframearea• Pageable1Mpageframes• AllocatedonSCM-capablemachines(zEC12/zBC12andabove)

• IfFlashExpressisinstalled,theselargepagesmaybepagedtoandfromSCM• IfFlashExpressisnotinstalled,thenifthosepagesareeverpagedout,theywillbedemotedto4Ksizepageframesand

willremain4KsizeuntilthenextIPL• System-definedsize

• Approximately12.5%ofonlinememory atIPLtime– adjustedtowhatfitsafterQuadandLFAreaarebuilt• Pageable1MframesoverflowintotheLFAREAwhenPLAreaisdepleted

34

Largesizerealmemorypageframes…• Let’sdosomemaths…

• Startingposition

• Ifyouweretoadd100GBtotheLPARanddefineitallasLFAREA

35

Onlinememory (GB) 150.0LFAREA(GB) 100.0QUAD(GB) 18.81MBPAGEABLE(GB) 18.84KBFRAMES(GB) 12.4

Onlinememory (GB) 50.0LFAREA(GB) 0QUAD(GB) 6.31MBPAGEABLE(GB) 6.34KBFRAMES(GB) 37.4

DonotforgetthatQuadareaandPageable 1MareagrowproportionallywithadditionalREALmemory!

Probablynotenough4Kframestohandlethe4Kworkloadneeds,includingtakingdumpsquickly,withouthavingtobreakdownfree1Mframes

Largesizerealmemorypageframes…• DB2canexploitfixed1Mand2Gpageframesforpage-fixedbufferpools

• Requirements• BufferpoolsmustbedefinedasPGFIX=YES• 1MsizepageframesrequiresDB210,z10andabove• 2GsizepageframesrequiresDB211,zEC12andabove

• DB211supportsFRAMESIZEparameter(4K,1M,2G)attheBPlevelforflexibility• DB211canalsoexploitfixed1Mpageframesforthelogoutputbuffer(OUTBUFF)• DB2canexploitpageable1Mframesforbufferpoolscontrolblocks(PMBs)

• PMBsconsumeonly4-5%ofbufferpoolallocationsbuttheyarereferencedveryheavily• Thisiswherecustomersgetthemostbenefit• Requirements

• DB210withAPARPM85944orDB211• BufferpoolscanbedefinedaseitherPGFIX=YESorPGFIX=NO,andcanbebackedbyeither4K,1Mor2Gsizepageframes

36

Largesizerealmemorypageframes…• Estimating‘optimal’LFAREA

• Totalof• (SumofVPSIZE*pagesizefromcandidatelocalbufferpools)*1.05• Plus20MBforz/OSusage• Pluslogoutputbuffersize(OUTBUFF)ifrunningDB211• Plusnon-DB2usagee.g.,Javaheapsizes• PlusanyoverflowfromPLArea (PageableLargeArea)

• Recommendation• DefineLFAREAbasedonwhatyoucanaffordafterconsideringtotalrealmemorydemandsfor4Kframesabove

2G• Mustconsideroperatingsystemmemory needs

• RSMrequirementformemorymapping(approximately1/64totalonlinerealmemoryatIPL)• Systemaddressspacesmemoryusage(DB2,CICS,etc.)• Mustalsoincludeenoughspare4Ksizeframesfortakingdumpsquickly

37

Whathappensincaseofshortageof4Kframes?• 2typesof4Krequests

• Preferred requests• Usedbynon-swapable addressspaces,canbefixedlongtermà DB2usespreferredrequests

• Non-Preferred requests• Usedbyswapable addressspaces,usedshorttermandnotfixed

• Pageable1Mlargeframearea• Pageable1Mframescanbebrokendowntosatisfydemandfor4Kpreferredornon-preferredrequests

• Quadarea• Quadframescanbebrokendowntosatisfydemandfor4Knon-preferredrequestsonly• Breakingnewsforz/OS2.2only!

• Severalcustomershavereportedexcessivepagingaftermigrationtoz/OS2.2• AnalysisshowedanabundanceofavailableQuadframesthatarenotusedbyRSM• FixedbyAPAROA50945 (HIPER)

38

Whathappensincaseofshortageof4Kframes?…• LFAREAandINCLUDEMAFCoption

• Availableframecount(AFC)isusedtodeterminewhenstoragemanagementshouldbeginpagingframes• Pre-z/OSV2R2:

• Bydefault,AFCdoesnotincludeLFAREA1Mpages• 1Mfixedframesarepreservedforfixed1Mrequests• Incaseofshortageof4Kframes,pagingcanoccurevenwithanabundanceofavailable1Mfixedframes

• INCLUDE1MAFC onLFAREAparametermeansAFCincludes theLFAREA1Mpages• NewkeywordaddedbyAPAROA41968• 1Mfixedframescanbebrokendowntosatisfydemandfor4Knon-preferredrequestsonly• Pagingisdelayed

• z/OSV2R2à TwooptionswhendefiningtheLFAREA:INCLUDEMAFC(YES|NO)• YES isthedefault

• RecommendationistouseINCLUDEMAFC(YES)sothatunused1Mpagescanbebrokendowntosatisfydemandfor4Knon-preferredrequests

39

Largesizerealmemorypageframes– Monitoring• Usefulcommands

• Findouthowmanyrealmemorypageframesofeachsizearebeingused• EspeciallyusefulwhenrunningmultipleDB2subsystemsonthesameLPAR

• DB210-DISPLAYBUFFERPOOL(BPx)SERVICE(4)• SeeDSNB999Imessage

• DB211-DISPLAYBUFFERPOOL(BPx)DETAIL(*)• SeeDSNB546Imessages

40

DSNB546I +PDJ1 PREFERRED FRAME SIZE 1M793600 BUFFERS USING 1M FRAME SIZE ALLOCATED

DSNB546I +PDJ1 PREFERRED FRAME SIZE 1M2956400 BUFFERS USING 4K FRAME SIZE ALLOCATED

DSNB999I +D2V1 DSNB1DBP SERVICE( 4 ) OUTPUT DSNB999I +D2V1 4K PAGES 0 DSNB999I +D2V1 1M PAGES 1476

Largesizerealmemorypageframes– Monitoring…• Usefulcommands…

• MVSDISPLAYVIRTSTOR,LFAREA• ShowtotalLFAREA,allocationsplitacrossdifferentsizepageframes,whatisavailable• SeeIAR019I message

41

IAR019I 10.45.57 DISPLAY VIRTSTORSOURCE = 00TOTAL LFAREA = 7782M , 0GLFAREA AVAILABLE = 2025M , 0GLFAREA ALLOCATED (1M) = 5377MLFAREA ALLOCATED (4K) = 374MMAX LFAREA ALLOCATED (1M) = 5377MMAX LFAREA ALLOCATED (4K) = 526MLFAREA ALLOCATED (PAGEABLE1M) = 6MMAX LFAREA ALLOCATED (PAGEABLE1M) = 2031MLFAREA ALLOCATED NUMBER OF 2G PAGES = 0MAX LFAREA ALLOCATED NUMBER OF 2G PAGES = 0

IAR019I 10.48.56 DISPLAY VIRTSTORSOURCE = C2TOTAL LFAREA = 14336M , 0GLFAREA AVAILABLE = 0M , 0GLFAREA ALLOCATED (1M) = 14329MLFAREA ALLOCATED (4K) = 0MMAX LFAREA ALLOCATED (1M) = 14329MMAX LFAREA ALLOCATED (4K) = 0MLFAREA ALLOCATED (PAGEABLE1M) = 7MMAX LFAREA ALLOCATED (PAGEABLE1M) = 7MLFAREA ALLOCATED NUMBER OF 2G PAGES = 0MAX LFAREA ALLOCATED NUMBER OF 2G PAGES = 0

UndersizedLFAREAonresultingin(small)lostCPUsavings

OversizedLFAREAresultinginashortageof4Ksizeframesandexpensivebreakdownof1Msizelargeframes– mayalsoleadtoprematurepagingifINCLUDE1MAFC=NO

Appendix

MonitoringREAL/AUXmemory usage– MappingforreferenceIFCIDFIELD PMFIELD PDBCOLUMNNAME MEMU2DescriptionQW0225RL QW0225RL REAL_STORAGE_FRAME DBM1REALinusefor31and64-bitpriv(MB)QW0225AX QW0225AX AUX_STORAGE_SLOT DBM1AUXinusefor31and64-bitpriv(MB)QW0225HVPagesInReal SW225VPR A2GB_REAL_FRAME DBM1REALinusefor64-bitpriv(MB)QW0225HVAuxSlots SW225VAS A2GB_AUX_SLOT DBM1AUXinusefor64-bitpriv(MB)QW0225PriStg_Real SW225PSR A2GB_REAL_FRAME_TS DBM1REALinusefor64-bitprivw/oBP(MB)QW0225PriStg_Aux SW225PSA A2GB_AUX_SLOT_TS DBM1AUXinusefor64-bitprivw/oBP(MB)QW0225RL QW0225RL DIST_REAL_FRAME DISTREALinusefor31and64-bitpriv(MB)QW0225AX QW0225AX DIST_AUX_SLOT DISTAUXinusefor31and64-bitpriv(MB)QW0225HVPagesInReal SW225VPR A2GB_DIST_REAL_FRM DISTREALinusefor64-bitpriv(MB)QW0225HVAuxSlots SW225VAS A2GB_DIST_AUX_SLOT DISTAUXinusefor64-bitpriv (MB)QW0225ShrStg_Real SW225SSR A2GB_SHR_REALF_TS REALinusefor64-bitshared(MB)QW0225ShrStg_Aux SW225SSA A2GB_SHR_AUXS_TS AUXinusefor64-bitshared(MB)QW0225ShrStkStg_Real SW225KSR A2GB_SHR_REALF_STK REALinusefor64-bitsharedstack(MB)QW0225ShrStkStg_Aux SW225KSA A2GB_SHR_AUXS_STK AUXinusefor64-bitsharedstack(MB)QW0225ComStg_Real SW225CSR A2GB_COMMON_REALF REALinusefor64-bitcommon(MB)QW0225ComStg_Aux SW225CSA A2GB_COMMON_AUXS AUXinusefor64-bitcommon(MB)QW0225_REALAVAIL S225RLAV QW0225_REALAVAIL REALAVAIL(MB)(S)

43

Note:AllREAL/AUXstoragefieldsinIFCID225andOMPEperformancedatabaseareexpressedin4KBframesorslots–theyshouldbeconvertedtoMB(conversionisalreadydoneinMEMU2)

MonitoringREAL/AUXmemoryusage– BasedonOMPEPDB• StackedAREAgraph- oneforeachDB2member(onesheetperDB2member)

• StackedAREAgraph- oneforeachDB2member(onesheetpermember)

• Linegraph- oneforeachLPAR

44

(REAL_STORAGE_FRAME - A2GB_REAL_FRAME)*4/1024 ASDBM1_REAL_PRIV_31BIT_MB(A2GB_REAL_FRAME - A2GB_REAL_FRAME_TS)*4/1024 ASDBM1_REAL_PRIV_64BIT_BP_MBA2GB_REAL_FRAME_TS*4/1024 ASDBM1_REAL_PRIV_64BIT_XBP_MB(DIST_REAL_FRAME - A2GB_DIST_REAL_FRM)*4/1024 ASDIST_REAL_PRIV_31BIT_MBA2GB_DIST_REAL_FRM*4/1024 ASDIST_REAL_PRIV_64BIT_MBA2GB_COMMON_REALF*4/1024 ASREAL_COM_64BIT_MBA2GB_SHR_REALF_TS*4/1024 ASREAL_SHR_64BIT_MBA2GB_SHR_REALF_STK*4/1024 ASREAL_SHR_STK_64BIT_MB

(AUX_STORAGE_SLOT - A2GB_AUX_SLOT)*4/1024 ASDBM1_AUX_PRIV_31BIT_MB(A2GB_AUX_SLOT - A2GB_AUX_SLOT_TS)*4/1024 ASDBM1_AUX_PRIV_64BIT_BP_MBA2GB_AUX_SLOT_TS*4/1024 ASDBM1_AUX_PRIV_64BIT_XBP_MB(DIST_AUX_SLOT - A2GB_DIST_AUX_SLOT)*4/1024 ASDIST_AUX_PRIV_31BIT_MBA2GB_DIST_AUX_SLOT*4/1024 ASDIST_AUX_PRIV_64BIT_MBA2GB_COMMON_AUXS*4/1024 ASAUX_COM_64BIT_MBA2GB_SHR_AUXS_TS*4/1024 ASAUX_SHR_64BIT_MBA2GB_SHR_AUXS_STK*4/1024 ASAUX_SHR_STK_64BIT_MB

QW0225_REALAVAIL*4/1024 ASREAL_AVAIL_LPAR_MB

OMPEspreadsheetgeneratorSampleJCL

45

//******************************************//* JOBCARD//******************************************//JOBLIB DD DISP=SHR,DSN=<DB2OMPE>.TKANMOD //*//STATFIL EXEC PGM=FPECMAIN //INPUTDD DD DISP=SHR,DSN=<USER_HLQ>.SMF100 //SYSPRINT DD SYSOUT=* //SYSOUT DD SYSOUT=* //JOBSUMDD DD SYSOUT=* //STFILDD DD DISP=(NEW,PASS),UNIT=SYSDA,SPACE=(CYL,(200,100)) //SYSIN DD * GLOBAL

INCLUDE (SUBSYSTEMID(DB2A)) STATISTICS

FILE DDNAME(STFILDD) EXEC /* //FPEPCSVP EXEC PGM=FPEPCSV, // PARM='SSTG Y Y N , . <DB2OMPE>.TKO2SAMP' //SYSPRINT DD SYSOUT=* //FLDSEL DD DISP=SHR,DSN=<USER_HLQ>.FPECSVU(CSVSTO11) //INPUT DD DISP=(OLD,DELETE),DSN=*.STATFIL.STFILDD //OUTPUT DD DSN=<USER_HLQ>.MEMU2.CSV, // DISP=(NEW,CATLG,DELETE), // UNIT=SYSDA,SPACE=(CYL,(200,100),RLSE), // DCB=(RECFM=VB,LRECL=32756,BLKSIZE=32760) //*

JohnCampbellIBMDB2forz/OSDevelopmentcampbelj@uk.ibm.com

@gurudb2

Session9007Starter'sguidetoREALmemorymanagementwithDB2forz/OS