A scientific approach to XenApp farm sizing

HelgeKlein.com Home of

SetACL

A scientific approach toXenApp farm sizing

Helge Klein

HelgeKlein.com Home of

SetACL

Who isHelge Klein?

CTP, MVPAuthor of SetACL and Delprof2Independent consultant and developerArchitect of what later became Citrix Profile Management

HelgeKlein.com Home of

SetACL

talking about?What is he

Scientifically sound farm sizing methodologyHow to calculate farm capacity

HelgeKlein.com Home of

SetACL

Methodology

1. Determine capacity of existing farm2. Measure load and identify bottlenecks3. Calculate capacity of new farm

HelgeKlein.com Home of

SetACL

ExampleDescription Farm capacity

old (total)Farm capacity old (per user)

Factor Farm capacity new (per user)

Farm capacity new (total)

RAM for user sessions [MB] 630,000 315 2.0 630 1,575,000

CPU [normalized cores] 1,250 0.63 0.25 0.16 391

Hard disk [write IOPS] 7,300 3.65 1.5 5.48 13,688

Network [MB/s] 24,000 12.00 0.1 1.20 3,000

HelgeKlein.com Home of

SetACL

Where to get the

numbers?1. Data collection2. Observation3. Measurements4. Calculation

HelgeKlein.com Home of

SetACL

Data collection

HelgeKlein.com Home of

SetACL

for /f %i in (AllFarmServers.txt) do wmic /node:%i cpu get name, maxclockspeed, systemname, description, manufacturer, revision /format:csv >> CPUs.txt

Collect CPU dataCreate AllFarmServers.txt with qfarm

Use resulting list to determine server model

HelgeKlein.com Home of

SetACL

Srv001,x86 Family 15 Model 4 Stepping 10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001Srv001,x86 Family 15 Model 4 Stepping 10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001Srv001,x86 Family 15 Model 4 Stepping 10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001Srv001,x86 Family 15 Model 4 Stepping 10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv002,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv003,x86 Family 6 Model 44 Stepping 2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003Srv004,x86 Family 6 Model 26 Stepping 5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004Srv004,x86 Family 6 Model 26 Stepping 5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004Srv004,x86 Family 6 Model 26 Stepping 5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004...

The result

HelgeKlein.com Home of

SetACL

RAM, NICshard disks

Could be determined via WMI, tooOften knowing the server model is sufficientComponents per model often identical

HelgeKlein.com Home of

SetACL

Component HP DL360 G4

CPU 2x Xeon dual core 3,0 GHz, no Hyper Threading4 Cores, CPUID: x86 Family 15 Model 4 Stepping 1

RAM 4 GB

NICs 2x Gigabit, 1 connected

RAID Smart Array 6i

Hard disks 2x 70 GB configured as RAID-1, Compaq BF0728A4CB, 3,5“, 15.000 R/Min, Ultra320 SCSI

Spec sheet

HelgeKlein.com Home of

SetACL

Resulting dataServer model NumberDL360 G7 26DL360 G6 52DL360 G5 70DL360 G4p 60DL360 G4 35

HelgeKlein.com Home of

SetACL

Observation

HelgeKlein.com Home of

SetACL

How many users are logged on?Load of CPU, RAM, NICs

Individual processes with a lot of RAM or CPU?

HelgeKlein.com Home of

SetACL

Physical Disk\% Disk Time

Physical Disk\Avg. Disk Queue Length

„Time“ is equivalent to flickering of hard disk LEDDisk queue length: Number of waiting IOs

HelgeKlein.com Home of

SetACL

Hypothesis: farm is

memory limited

Limiting factor will differ between farms

HelgeKlein.com Home of

SetACL

Measurements

Tool: PerfmonNext slides: relevant counters

HelgeKlein.com Home of

SetACL

Terminal Services\Active SessionsTerminal Services\Inactive Sessions

Terminal Services\Total SessionsSystem\Processes

General system information

HelgeKlein.com Home of

SetACL

PhysicalDisk(_Total)\% Disk TimePhysicalDisk(_Total)\Avg. Disk Queue Length

PhysicalDisk(_Total)\Disk Reads/secPhysicalDisk(_Total)\Disk Writes/sec

PhysicalDisk(_Total)\Avg. Disk sec/Transfer

Hard disk activityLoad, queue length, operations per second, latency

HelgeKlein.com Home of

SetACL

Processor(_Total)\% Processor TimeMemory\Available MBytes

Network Interface(*)\Bytes Total/sec

CPU, RAM and networkRAM: total amount must be known!

HelgeKlein.com Home of

SetACL

logman create counter TSPerf -f csv -cf C:\PerfLogs\Counters.txt -o C:\PerfLogs\Server13.csv -si 60 -rf 24:00:00

AutomationCreate and start data collector set. Format CSV, performance counters are

read from C:\PerfLogs\Counters.txt, output file is C:\PerfLogs\Server013.csv, 60 second sampling interval, duration 24 hours.

HelgeKlein.com Home of

SetACL

logman create counter TSPerf -f csv -cf C:\PerfLogs\Counters.txt –o C:\PerfLogs\Server13.csv -si 60 -rf 24:00:00-s Server13

Execution on remote computer Server13

HelgeKlein.com Home of

SetACL

for /f %i in (Servers.txt) do logman create counter TSPerf -f csv -cf C:\PerfLogs\Counters.txt -o C:\PerfLogs\%i.csv -si 60 -rf 24:00:00 -s %i

Many serversOne computer name per line in Servers.txt

HelgeKlein.com Home of

SetACL

Analyzingthe measured data

HelgeKlein.com Home of

SetACL

CPU and sessions

Server 37

Moderate load during logon phase,

afterwards even less

A lot of overcapacity

HelgeKlein.com Home of

SetACL

CPU and sessions

Server 89

HelgeKlein.com Home of

SetACL

Overlaying the CPU load of

many servers

Easily verify the analyis

HelgeKlein.com Home of

SetACL

HDD and sessions

Server 37

Moderate load, peaks during logon phase

Full load at aprox. 200

HelgeKlein.com Home of

SetACL

HDD and sessions

Server 89

HelgeKlein.com Home of

SetACL

Overlaying the HDD load of

many servers

Easily verify the analyis

HelgeKlein.com Home of

SetACL

RAM and sessions

Server 37

Continually increasing load, maximum in the

afternoon

Available RAM must not go near zero

(because of disk cache)

High load

HelgeKlein.com Home of

SetACL

RAM and sessions

Server 89

HelgeKlein.com Home of

SetACL

Overlaying the memory load

of many servers

Easily verify the analyis

HelgeKlein.com Home of

SetACL

Network and sessions

Server 37

200 = 2 MB/s

Average rate < 200 KB/s

Very low load, a lot of overcapacity

HelgeKlein.com Home of

SetACL

Network and sessions

Server 89

HelgeKlein.com Home of

SetACL

Overlaying the network load

of many servers

Easily verify the analyis

HelgeKlein.com Home of

SetACL

Hypothesis confirmed: farm is limited by

available memoryCPU load: low, network: negligiblehard disk: moderate

HelgeKlein.com Home of

SetACL

Calculatingfarm capacity

Description Farm capacity old (total)

Farm capacity old (per user)

Factor Farm capacity new (per user)

Farm capacity new (total)

RAM for user sessions [MB] 630,000 315 2.0 630 1,575,000

CPU [normalized cores] 1,250 0.63 0.25 0.16 391

Hard disk [write IOPS] 7,300 3.65 1.5 5.48 13,688

Network [MB/s] 24,000 12.00 0.1 1.20 3,000

HelgeKlein.com Home of

SetACL

How to compare performance of different CPUs?Benchmarking is difficultBetter: Moore’s law (doubling of performance every 18-24 months)Surprisingly accurate (amongst other things because it is a self-fulfilling prophecy)

NormalizingCPU performance

HelgeKlein.com Home of

SetACL

Performance after time (in months)Assumed performance doubling every 21 monthsOldest CPU in farm = 1,0

𝐿(𝑡 )=𝑒0,033𝑡

HelgeKlein.com Home of

SetACL

CPU Cores First sold Performance index Performance index per core

Xeon 7100 „Tulsa“ 3,0 GHz

2 08/2006 1,0 1,0

Xeon 7100 „Tulsa“ 3,4 / 3,6 GHz

2 03/2007 1,26 1,26

Xeon E5430 / E5440

4 11/2007 1,64 0,82

Xeon X5550 4 03/2009 2,78 1,39

Xeon E5640 4 03/2010 4,13 2,06

HelgeKlein.com Home of

SetACL

CPU Number of CPUs

Number of cores

Number of normalized CPUs

Number of normalized cores

Xeon 3 GHz 70 140 70 140

Xeon 3,4/3,6 GHz

119 238 150 300

Xeon E5430/E5440

70 280 115 230

Xeon X5550 67 268 186 370

Xeon E5640 26 104 107 210

Total 352 1030 628 1250

HelgeKlein.com Home of

SetACL

Farm capacity:

1250 normalized CPUs

0.63 CPUs / user

HelgeKlein.com Home of

SetACL

Hard disk performance =

IOPSWith many concurrent accesses transfer rate is mostly irrelevantMore important: IOPS (operations per second)Exact number depends on measurement method do not believe vendors

HelgeKlein.com Home of

SetACL

PhysicalDisk(_Total)\Disk Reads/secPhysicalDisk(_Total)\Disk Writes/sec

IOPS measurement with PerfmonRead and write IOPS may be very different

HelgeKlein.com Home of

SetACL

Read and write IOPS

Average: ~15

Including spikes: 30

HelgeKlein.com Home of

SetACL

Farm capacity:

7300 IOPS

3,7 IOPS / user

HelgeKlein.com Home of

SetACL

RAM – how muchdo we have?

We need total RAM that is available for user sessions

HelgeKlein.com Home of

SetACL

Server model RAM / server [GB] #Servers RAM / server total [GB]

DL360 G4 3,5 35 122,5

DL360 G4p 3,5 60 210

DL360 G5 3,25 70 227,5

DL360 G6 3,49 52 181,48

DL360 G7 3,49 26 90,74

Total 243 832

HelgeKlein.com Home of

SetACL

Description Memory [MB]Session 0 (OS + Presentation Server + antivirus) 325

Kernel (pooled + non-pooled) 175File system cache 400Total 900

Calculate RAM used by the OS (Server 2003)

HelgeKlein.com Home of

SetACL

Farm capacity:

630 GB

315 MB / user

25% Overhead because 900 MB used by OS

HelgeKlein.com Home of

SetACL

Network – too much for HDX?

1 Gigabit/second per server~ 100 MB/s

HelgeKlein.com Home of

SetACL

Farm capacity:

24 GB/s

12 MB/s / user

Extreme overcapacity

HelgeKlein.com Home of

SetACL

Futurecapacity

Description Farm capacity old (total)

Farm capacity old (per user)

Factor Farm capacity new (per user)

Farm capacity new (total)

RAM for user sessions [MB] 630,000 315 2.0 630 1,575,000

CPU [normalized cores] 1,250 0.63 0.25 0.16 391

Hard disk [write IOPS] 7,300 3.65 1.5 5.48 13,688

Network [MB/s] 24,000 12.00 0.1 1.20 3,000

HelgeKlein.com Home of

SetACL

Parameters. Calculation includes:Changed user count, different memory requirements of new OS,server replacement, capacity increase for disaster recovery

Parameter ValueFarm users old (CCU) 2,000

Farm users new (CCU) 2,500

RAM for OS new [MB] 1,300

CPU performance new [normalized cores]

1.9

IOPS per disk new 300

Reserve for disaster recovery 1.5

HelgeKlein.com Home of

SetACL

Servers are memory limited after setting the amount of RAM,the other parameters can be calculated

Virtual server

RAM [MB] 32.768

User sessions 49.9

CPU cores (physical) 4.1

IOPS 273

NICs 0.6

HelgeKlein.com Home of

SetACL

After setting the number of VMs per host the other parameterscan be calculated.Excel sheet available upon request.

Physical serversVMs per host 3 #Virtual

servers50

RAM [MB] 101,376 #Hosts 25

CPU cores 12 Users / host

150

Hard disks 3

NICs 2

HelgeKlein.com Home of

SetACL

Thank you!Helge Klein