Disk I/O PerformanceDisk I/O Performance

Emphasizing Caching and XFCEmphasizing Caching and XFC

Ian PercivalIan PercivalIan PercivalIan Percival XFC Project Leader OpenVMS Engineering Ian.percival@compaq.com

Application I/O PerformanceApplication I/O PerformanceApplication I/O PerformanceApplication I/O Performance

Can you improve your I/O performance? Best performance comes from not doing I/O at all,

otherwise get as “close” to the application as you can. Application level considerations. Operating System Constraints and Considerations. Hardware Constraints and considerations.

Hardware ConsiderationsHardware ConsiderationsHardware ConsiderationsHardware Considerations– Disk Types and caches

DecRam 15000, 10000, 7200 rpm, solid state Caches disabled on local SCSI

– Disk Controllers and caches Writethrough, Writeback, Memory

– RAID level– Interconnects

Latency and throughput

– CPU and I/O configuration SMP NUMA

Software ConfigurationSoftware ConfigurationSoftware ConfigurationSoftware Configuration File system

– Fragmentation RMS

– Tuning– Local and Global Buffers

XFC Raid Fastpath MSCP

Tools for analysisTools for analysisTools for analysisTools for analysis

Application itself– Transaction monitor, etc

Availability Manager SHOW and MONITOR SDA Third Party products

Minimize I/O in the first placeMinimize I/O in the first placeMinimize I/O in the first placeMinimize I/O in the first place

Good application design Application knowledge of “hot files” should use

global and local buffers. Try to satisfy I/O as close to the application as

possible, by using caches – software and hardware.

Software HierarchySoftware HierarchySoftware HierarchySoftware Hierarchy

Disk DriverDisk Driver

RMSRMS

File System (XQP)File System (XQP)

ApplicationApplication

eXtended File CacheeXtended File CacheeXtended File CacheeXtended File Cache

Replaces VCC in V7.3 Scales - 64 bit cache can be very large No closed file limit I/Os larger than 35 blocks Readahead where appropriate

Sample Commercial ProgramSample Commercial ProgramSample Commercial ProgramSample Commercial Program

Real world case study I/O intensive Financial Program ES40 4 CPUs Local 10,000 rpm SCSI, 2 volume

set 65% reads. Approximately 7.5 Million I/Os. Clustered, but only a Single Node is performing

the job.

Elapsed TimesElapsed TimesElapsed TimesElapsed Times No Cache or extra RMS buffers

– 43 Hours 28 Mins 45 Secs VCC (the old data cache)

– 8 Hours 15 Mins 22 Secs Out of the box V7.3 using XFC

– 7 Hours 11 Mins 16 Secs Use RMS Global Buffers on 10 “hot files” with no cache

– 12 Hours 4 Mins 55 Secs XFC and RMS Global buffers

– 7 Hours 11 Mins 55 Secs

RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC? RMS Global buffers WILL cache files open for writeshare

in a cluster XFC will not in V7.3 Obviously RMS buffers require knowledge of IO to apply

them effectively. Can use the SET FILE/GLOBAL_BUFFER=n command to enable them once you know which files are “hot”.

XFC is S2 Space based and has no effective limit to its size other than physical memory available

RMS Global buffers are currently limited to a max value of 32767 buffers per file.

RMS Global Buffers AND XFC make a RMS Global Buffers AND XFC make a good pairgood pairRMS Global Buffers AND XFC make a RMS Global Buffers AND XFC make a good pairgood pair

As RMS is closer to the User/App it always makes sense to use it.

Must use global buffers for files open for shared write in a cluster – RIGHTSLIST.DAT is a good example here on a common system disk.

XFC will cache as much as it can once RMS passes I/O to it

Hot Files and Data AccessesHot Files and Data AccessesHot Files and Data AccessesHot Files and Data Accesses

All Disk I/O flows through XFC Enhancement to DCL SHOW MEM

– SHOW MEMORY/CACHE=

– (VOLUME=PICKLE, TOPQIO=20, OPEN, CLOSED)

– Limitation: Only works for Virtual I/O Can also use various third party products

HotFiles CommandHotFiles CommandHotFiles CommandHotFiles Command

$ show memory/cache=(volume=iss$,topqio=2,open,closed) System Memory Resources on 29-SEP-2000 10:09:56.86

Extended File Cache Top QIO File Statistics:

_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible

_$9$DKB500:[iSS$ROOT.iSS$.APPFILES6]COBEGOP.PLO;72 (open) Caching is enabled, active caching mode is Write Through Allocated pages 7664 Total QIOs 2650066 Read hits 2122691 Virtual reads 2133065 Virtual writes 517001 Hit rate 80 % Read aheads 0 Read throughs 2133065 Write throughs 517001 Read arounds 0 Write arounds 0

I/O and Cache Statistics for the VolumeI/O and Cache Statistics for the VolumeI/O and Cache Statistics for the VolumeI/O and Cache Statistics for the Volume

$show memory/cache=volume=iss$ System Memory Resources on 29-SEP-2000 10:27:23.36

Extended File Cache Volume Statistics:

_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible Open files 115 Closed files 110 Files ever opened 1309 Files ever deposed 1084 Allocated pages 143512 Locks acquired 1198 Total QIOs 4425333 Read hits 3211408 Virtual reads 3366252 Virtual writes 1056131 Hit rate 72 % Read aheads 6638 Read throughs 3365919 Write throughs 1056131 Read arounds 333 Write arounds 0 Ave Disk I/O Resp Time incl cache hits (microseconds) 3107

SHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDE

show mem/cache

System Memory Resources on 29-SEP-2000 11:06:14.21

Extended File Cache (Time of last reset: 28-SEP-2000 11:29:35.85) Allocated (Mbytes) 3000.00 Maximum size (Mbytes) 5120.00 Free (Mbytes) 976.38 Minimum size (Mbytes) 3000.00 In use (Mbytes) 2023.61 Write hit rate 0% Read hit rate 93% Write I/O count 2731285 Read I/O count 5346002 Write hit count 0 Read hit count 5012771 Writes bypassing cache 0 Reads bypassing cache 337 Vols in Full XFC mode 0 Files cached open 522 Vols in VIOC Compatible mode 3 Files cached closed 478 Vols in No Caching mode 0 Percentage Read I/Os 66% Vols in Perm. No Caching mode 0

SHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULL I/O Statistics - Distributions (MAX_IO_SIZE: 127) ------------------------------------------------- Transfer Size: Reads Read Hits Writes 1 Block IO: 472510 472446 115923 2 Block IO: 7553 7481 426 3 Block IO: 2068434 2035932 753483 4 Block IO: 199136 172784 324110 5 Block IO: 27315 22481 50520 6 Block IO: 1613335 1475185 1205452 7 Block IO: 109592 107681 155111 8 Block IO: 16076 16041 146 9 Block IO: 748511 620923 117591 10 Block IO: 78 67 165 11 Block IO: 884 879 8 12 Block IO: 7572 7563 3314 13 Block IO: 3333 3329 6 14 Block IO: 56 48 13 15 Block IO: 1487 1486 10 16 Block IO: 8900 7870 2086 17 Block IO: 374 368 9 18 Block IO: 21514 21153 2841 19 Block IO: 60 54 5 20 Block IO: 17 13 0

Per Volume and Per File IO HistogramPer Volume and Per File IO HistogramPer Volume and Per File IO HistogramPer Volume and Per File IO Histogram Not there by default Will provide a utility to enable this

– (not there as standard in V7.3) Per file and per volume Statististics Resets are part of this SET CACHE/RESET

More on Response TimesMore on Response TimesMore on Response TimesMore on Response Times

Many third party products use Little’s Law As does Compaq’s Capacity Planner – Little’s Law can be

manipulated to give:– Response time = average queue length/IO rate

Not too useful for lower I/O Volumes where queues don’t build up. XFC performs an RSCC() call when an I/O enters it. It performs another RSCC() when the I/O is about to complete (User

process is aboutto be awakened again) SMP accuracy limitation RMS buffer hits are not included

How to get more on response timesHow to get more on response timesHow to get more on response timesHow to get more on response times

Use XFC SDA extension for more detail– $ ANALYZE/SYSTEM

– SDA> XFC SHOW VOLUME/BRIEF

– SDA> XFC SHOW VOLUME This is the PER NODE view – need to do this clusterwide

to get the entire picture

XFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFSDA> xfc show volume/brief

Summary of XFC Cached Volumes (CVBs)

------------------------------------

Name: DISK$ISS_V2 CVB: FFFFFFFE495FF810 Cached Files: 000000000000000026 (

38)

Name: DISK$ISS_V1 CVB: FFFFFFFE495FFA70 Cached Files: 000000000000000073 (

115)

Name: DISK$X73W_DENBO2 CVB: FFFFFFFE495FFCD0 Cached Files: 000000000000000174 (

372)

Volumes found: 3

XFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetCached Open Files: 38

Cached Closed Files: 8

Files Ever Opened: 160

Files Ever Deposed: 114

Pages Allocated: 97270

Total QIOs: 4248036

Read Hit Count: 2372227

Virtual Read Count: 2560160

Virtual Write Count: 1687876

Read Percentage: 60 %

Hit Rate: 55 %

Average Overall I/O response time to this Volume

in microseconds: 4298

Average Cache Hit I/O response time to this Volume

in microseconds: 24

Average Disk I/O response time to this Volume

in microseconds: 9703

Accuracy of I/O resp time: 70 %

Readahead Count: 50

XFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetCached Open Files: 115

Cached Closed Files: 110

Files Ever Opened: 1309

Files Ever Deposed: 1084

Pages Allocated: 143517

Total QIOs: 4431899

Read Hit Count: 3217640

Virtual Read Count: 3372484

Virtual Write Count: 1056301

Read Percentage: 76 %

Hit Rate: 72 %

Average Overall I/O response time to this Volume

in microseconds: 3103

Average Cache Hit I/O response time to this Volume

in microseconds: 21

Average Disk I/O response time to this Volume

in microseconds: 11262

Accuracy of I/O resp time: 82 %

Readahead Count: 6720

RMS Global BuffersRMS Global BuffersRMS Global BuffersRMS Global Buffers

SET FILE/STATISTICS to enable monitoring Elinor did a super job by allowing the /SHARE qualifier to let you do

this live. For example:

– SET FILE/STAT/SHARE HALLOWEEN.DAT– MONITOR RMS /ITEM=CACHING/FILE=HALLOWEN.DAT– SET FILE/NOSTAT/SHARE … turns it off

Global Buffers use global sections and hence global pagefile space.

XFC and memory usage.XFC and memory usage.XFC and memory usage.XFC and memory usage.

In general, XFC (or any cache) will be more effective the more main memory you have.

Operates in Dynamic or Permanent or both modes. Permanent memory is assigned by using RESERVED MEMORY. Probably a good idea to reserve some percentage of your memory

for XFC, leaving the remainder dynamic, assuming your system isn’t memory starved.

NUMA support.– RESERVED MEMORY only

XFC SYSGEN ParamersXFC SYSGEN ParamersXFC SYSGEN ParamersXFC SYSGEN Paramers

VCC_MAX_CACHE– Default is -1– (Note not VCC_MAXSIZE – VIOC only)

VCC_MAX_IO_SIZE– Default is 127

(VIOC was hard coded to 34 blocks)

– Histograms should show when you need to increase this

XFC Reserved MemoryXFC Reserved MemoryXFC Reserved MemoryXFC Reserved Memory

SYSMAN registry SYSMAN> RESERVED_MEMORY ADD

VCC$MIN_CACHE_SIZE/ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL/SIZE=xx

– Size is in units of Mb For NUMA support, you should add /RAD=xx with the above

command, entering it for each RAD you wish to use. XFC Dynamic memory is NOT currently RAD selectable.

SHOW MEMORY with XFC ReservationSHOW MEMORY with XFC ReservationSHOW MEMORY with XFC ReservationSHOW MEMORY with XFC Reservation

$ SHOW MEMORY/RES

Memory Reservations (pages): Group Reserved In Use Type

VCC$MIN_CACHE_SIZE --- 384000 384000 Allocated

Total (2.92 Gbytes reserved) 384000 384000

Don’t Cache unimportant devicesDon’t Cache unimportant devicesDon’t Cache unimportant devicesDon’t Cache unimportant devices

In general, systems are not going to be memory rich. Caching many Tb of disk space with just a few Gb of physical memory may not give high hit rates.

MOUNT/NOCACHE – Disables XQP caches as well as VCC/XFC

For individual control– SET FILE/CACHE=NO_CACHING X.X

Extract of DIR/FULL commandExtract of DIR/FULL commandExtract of DIR/FULL commandExtract of DIR/FULL command $ dir/full sys$system:sysdump.dmp

Directory SYS$SYSROOT:[SYSEXE]

SYSDUMP.DMP;1 File ID: (6365,2,0) Size: 350000/350010 Owner: [1,1] Created: 4-AUG-1999 14:25:12.78 Revised: 29-SEP-2000 12:19:11.22 (66) Expires: <None specified> File organization: Sequential Shelved state: Online Caching attribute: No_caching

ReadaheadReadaheadReadaheadReadahead

XFC WILL by default attempt to readahead where it deems it to be appropriate.

VCC_READAHEAD controls this The SHOW MEMORY/CACHE=VOLUME command gives the

number of readaheads. Enabled/Disabled on a system wide basis, if disk queue lengths are

already high, and if cache misses occur on the readaheads, may want to turn it off.

XFC/VCC are client basedXFC/VCC are client basedXFC/VCC are client basedXFC/VCC are client based

Will NOT cache served I/O – MSCP served or QIOServer Served – on the Server.

Third Party products do exist which are LBN caches. Some time in the future QIOServer may implement

an LBN Cache.

Using SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETC

Volatility is again an issue. Most modern disk devices have on disk caching

ability. If have recovery mechanisms, then could consider

this. Benchmark above which took 7 hours 11 Mins now

takes just 5 Hours 25 Minutes 04 Secs – though of course there is now a risk component, if the power was to fail.

Local Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on Disk

show mem/cache=vol=iss$

System Memory Resources on 1-OCT-2000 08:26:43.52

Extended File Cache Volume Statistics:

_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible Open files 115 Closed files 110 Files ever opened 1311 Files ever deposed 1086 Allocated pages 143548 Locks acquired 1200 Total QIOs 4433091 Read hits 3218148 Virtual reads 3372990 Virtual writes 1056810 Hit rate 72 % Read aheads 6728 Read throughs 3372655 Write throughs 1056810 Read arounds 335 Write arounds 0 Ave Disk I/O Resp Time incl cache hits (microseconds) 2217

Writeback CachingWriteback CachingWriteback CachingWriteback Caching

Not yet implemented Future enhancement for XFC On a per file, directory or Volume basis

– SET FILE/CACHE=NO_CACHING– SET FILE/CACHE=WRITETHROUGH– SET FILE/CACHE=WRITEBACK

For application writers, $FLUSH command – equivalent of SYNCH in Unix file systems.

Ian PercivalIan PercivalIan PercivalIan Percival XFC Project Leader OpenVMS Cluster Engineering Ian.percival@compaq.com