performancemonitoring-100409184324-phpapp01

7/29/2019 performancemonitoring-100409184324-phpapp01

1/98

Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware

products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

Developing Performance Monitoring Tools UsingvSphere APIs: Techniques and Best Practices

TEXIBP 1005

Balaji Parimi & Ravi Soundararajan, VMware Inc.


2/98

2Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware



products are covered by one or more patents listed at htt

p://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

This session may contain product features that are currently underdevelopment.

This session/overview of the new technology represents nocommitment from VMware to deliver these features in any generally

available product.

Features are subject to change, and must not be included incontracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or featuresdiscussed or presented have not been determined.

These features are representative of feature areas under development. Feature commitments

are subject to change, and must not be included in contracts, purchase orders, or salesagreements of any kind. Technical feasibility and market demand will affect final delivery.

Disclaimer


3/98




products are covered by one or more patents listed at htt

p://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

t

LinkedinSearch forVMware Partner Network

Follow Us

TwitterSearch forVMware_Partners

FacebookSearch forVMware Partner Network

http://blogs.vmware.com/powerofpartnership/

Help us help you! Please fill out the surveyat the end of the course.

Join the conversationGet the latest scoopBe a part of the network

Subscribe to RSS

Follow this event LIVE!#VMwarePEX2010


4/98





Who is our Target Audience?

System Administrators: Monitor the performance of their virtual infrastructure

Understand the bottlenecks in the datacenter to help reconfigure their storageor network topology

Plan for future growth in the datacenter

VMware ISV Partners:

Collect the relevant/important performance counters and hand them off to aperformance analysis tool

Create a wizard/tool that helps system administrators troubleshoot theirdatacenter


5/98





Agenda

Use Cases

Building a Performance Monitoring Application

Q & A


6/98





Use Cases

Use Cases: What do I look for and why? CPU

Memory

Disk

Network


7/98





CPU

ESX is designed to run Virtual MachinesSchedulable entity = world

Virtual Machines are composed of worlds (mks, vCPUs)

Service Console (agents like vpxa, hostd) (Classic ESX)

Proportional-share scheduler for resource management Limits, Shares, and Reservations

World states (simplified view):

ready = ready-to-run but no physical CPU free

run = currently active and running

wait = blocked on I/O


8/98





So, How Do I Spot CPU Performance Problems?

One common issue is high CPU ready time High ready time possible contention for CPU resources among VMs

Many possible reasons

CPU over commitment (high %rdy + high %used)

Workload variability

Limit set on VM

No fixed threshold, but > 20% for a VCPUInvestigate further


9/98





CPU: Useful Metrics

Per-Host

Metric (Client) Metric

(esxtop)

Metric (sdk) Description

Usage (%) %USED cpu.usage.average CPU used overthe collectioninterval (%)

Usage (MHz) n/a cpu.usagemhz.average CPU used overthe collectioninterval (MHz)


10/98





CPU: Useful Metrics

Per-VM

Metric (Client) Metric(esxtop)

Metric (SDK) Description

Usage (%) %USED cpu.usage.average CPU used overthe collectioninterval

Used (ms) %USED cpu.used.summation CPU used overthe collectioninterval)*

Ready (ms) %RDY cpu.ready.summation CPU time spentin ready state*

Swap wait time(ms) [ESX4.0

hosts]

%SWPWT cpu.swapwait.summation CPU time spentwaiting for host-

level swap-in

* Units different between esxtop and vSphere client


11/98





Spotting CPU Overcommitment in esxtop

2-CPU box, but 3 active VMs (high %used)

High %rdy + high %used can imply CPU overcommitment


12/98





Caveat on Ready Time: Workload Variability

Ready time < used time

Used time

Ready time ~ used time

Some caveats on ready time

Used time ~ ready time: may

signal contention. However,might not be overcommitteddue to workload variability

In this example, we haveperiods of activity and idleperiods: CPU isntovercommitted all the time

(screenshot from VI Client)


13/98





Further ready time examination

High Ready TimeHigh MLMTD: there is a limit on this VM

High ready time not always because of overcommitmentWhen you see high ready time, double-check if limit is set


14/98





Further Ready Time Investigation: No Fixed ThresholdWhy?

ESX 1 ESX 2

vSphereVM

Probe VM vSphereDB

ODBC

Sniff

traffic

vSphere communicates with DBProbe VM monitors vSphere-to-DB traffic

The more traffic, the more work done by Probe VM


15/98





Why Isnt There A Fixed Ready Time Threshold?

Ready time jump from 12.5% (idle DB) to 20% (busy DB) didntnotice until responsiveness suffered!


16/98





Summary of Possible Reasons for High Ready Time

CPU overcommitment Possible solution: add more CPUs or VMotion the VM

Workload variability

A bunch of VMs wake up all at once

Note: system may be mostly idle: not always overcommitted

Limit set on VM

4x2GHz host, 2 vcpu VM, limit set to 1GHz (VM can consume 1GHz)

Without limit, max is 2GHz. With limit, max is 1GHz (50% of 2GHz)

CPU all busy: %USED: 50%; %MLMTD & %RDY = 150% [total is 200%, or 2CPUs]


17/98





Where do we go from here?

Use Cases: What do look for and why CPU

Memory

Disk

Network


18/98





Memory

ESX must balance memory usage for all worlds Virtual machines, Service Console, and vmkernel consume memory

Page sharing to reduce memory footprint of Virtual Machines

Ballooning to relieve memory pressure in a graceful way

Host swapping to relieve memory pressure when ballooning insufficient

ESX allows overcommitment of memory

Sum of configured memory sizes of virtual machines can be greater thanphysical memory if working sets fit


19/98





Ballooning vs. Swapping (1)

Ballooning: Memctl driver grabs pages and gives to ESX Guest OS choose pages to give to memctl (avoids hot pages if possible): either

free pages or pages to swap

- Unused pages are given directly to memctl

- Pages to be swapped are first written to swap partition within guest OS and then given to

memctl

VM1

Swap partition w/inGuest OS

ESX

VM2

memctl

1. Balloon

2. Reclaim

3. Redistribute

F


20/98






Swapping: ESX reclaims pages forcibly Guest doesnt pick pagesESX may inadvertently pick hot pages (possible VM

performance implications)

Pages written to VM swap file

VM1

SwapPartition(w/in guest)

ESX

VM2

VSWP(external to guest)

1. Force Swap2. Reclaim

3. Redistribute


21/98






Bottom line: Ballooning may occur even when no memory pressure just to keep memory

proportions under control

Ballooning is vastly preferably to swapping

Guest can surrender unused/free pages- With host swapping, ESX cannot tell which pages are unused or free and may accidentally pick

hot pages

Even if balloon driver has to swap to satisfy the balloon request, guest chooses what

to swap

- Can avoid swapping hot pages within guest


22/98





Ok, So Why Do I Care About Memory Usage?

If running VMs consume too much host memory Some VMs do not get enough host memory

This forces either ballooning or host swapping to satisfy VM demands

Host swapping or excessive ballooning reduced VM performance

If I do not size a VM properly (e.g., create Windows VM with 128MBRAM)

Withinthe VM, swapping occurs, resulting in disk traffic

VM may slow down Butdont make memory too big! (High overhead memory)


23/98

23Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMwareproducts are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMwareproducts are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

Important Memory Metrics (Per VM)

One rule of thumb: > 1MB/s swap in or swap out rate may meanmemory overcommitment


(esxtop)


Swap in rate

(ESX4.0 Hosts)

SWR/s mem.swapinRate.average Rate at which mem is

swapped in from disk

Swap out rate(ESX4.0 Hosts)

SWW/s mem.swapoutRate.average Rate at which mem isswapped out to disk

Swapped SWCUR mem.swapped.average (level 2counter)

~swap out swap in

Swap in(cumulative)

n/a mem.swapin.average Mem swapped in fromdisk

Swap out(cumulative)

n/a mem.swapout.average Mem swapped out todisk


24/98



Important Memory Metrics (Per Host, sum of VMs)

One rule of thumb: > 1MB/s swap in or swap out rate may meanmemory overcommitment


(esxtop)


Swap in rate

(ESX4.0 Hosts)

SWR/s mem.swapinRate.average Rate at which mem is

swapped in from disk

Swap out rate(ESX4.0 Hosts)

SWW/s mem.swapoutRate.average Rate at which mem isswapped out to disk

Swap used SWCUR mem.swapused.average (level2 counter)

~swap out swap in

Swap in(cumulative)

n/a mem.swapin.average Mem swapped in fromdisk

Swap out(cumulative)

n/a mem.swapout.average Mem swapped out todisk


25/98



Example of Swapping

Increased swap activity may be a sign of over-commitment

Lots of swapping

No swapping


26/98



A Stacked Chart (per VM) of Swapping

Lots ofswapping

No swapping


27/98




Use Cases: What do look for and why CPU

Memory

Disk

Network


28/98



Disk Performance Problems 101

What should I look for to figure out if disk is an issue? Am I getting the IOPs I expect?

Am I getting the bandwidth (read/write) I expect?

Are the latencies higher than I expect?

Where is time being spent?

What are some things I can do?

Make sure devices are configured properly (caches, queue depths)

Use multiple adapters and multipathing

Check networking settings (for iSCSI/NAS)


29/98



Useful Disk Metrics

Metric(Client)

Metric(Esxtop)


Commands CMDS/s disk.commands.summation commands

issued in thesampling interval

Read rate MBREADS/s disk.read.average KB/s read*

Write rate MBWRTN/s disk.write.average KB/s written*

Device latency DAVG/cmd disk.deviceLatency.average Average latencyat device

Kernel latency KAVG/cmd disk.kernelLatency.average Average latencyin vmkernel

Command

latency

GAVG/cmd disk.totalLatency.average Total latency for

command



30/98



Disk Performance Example: vSphere Client

SAN cache enabled:High Write Throughput

SAN cache disabled:Poor throughput


31/98



Another Disk Example: Slow VM Power On

Trying to Power on a VM Sometimes, powering on VM would take 5 seconds

Other times, powering on VM would take 5 minutes!

Where to begin?

Powering on a VM requires disk activity on host Check disk metrics for host


32/98



Lets look at the vSphere client

Max Disk Latencies range from 100ms to 1100msvery high! Why?(counter name: disk.maxTotalLatency.latest)

U i E E i Sl VM P O


33/98



Using Esxtop to Examine Slow VM Power On

Note very large DAVG/cmd and GAVG/cmd

Rule of thumb: GAVG/cmd > 50ms = high latency!

What does this mean?

Latency when command reaches device is high

Latency as seen by the guest is high

Low KAVG/cmd: command isnt queuing in VMkernel

Whats up?

Hi h Di k L t M t S l d


34/98



High Disk Latency: Mystery Solved

Host events: disk has connectivity issues high latencies!

Bottom line: monitor disk latencies; issues may not be related tovirtualization!

A M C l Di k E l


35/98



A More Complex Disk Example

ESX

VM1 VM2 VM3 VM4 VM15

NAS

Ethernet

Disk Traffic

VMs communicate with NAS (network traffic)NAS writes to disk (disk traffic)

Combining Metrics: A More Complex Disk Example


36/98



Combining Metrics: A More Complex Disk Example

Setup:

Group of Virtual Machines running on a host

Each Virtual Machine talks to a Virtual Machine serving as a NAS device

Problem:

Suddenly, I cannot log in to any of the Virtual Machines (really slow!)

Initial Speculation

Virtual Machines are saturated in some resource

A More Complicated Disk Example Part I (CPU)


37/98



A More Complicated Disk Example, Part I (CPU)

Predictable CPU usage,Host not saturated

Chaotic CPU usage,Host saturated

Complicated Disk Example Part 2: Disk Usage


38/98



Complicated Disk Example, Part 2: Disk Usage

Uneven, reduced disk usage

Predictable, balanced disk usage

Complicated Disk Example Part 3: Write Rate


39/98



Complicated Disk Example, Part 3: Write Rate

Increased write traffic, zero read traffic

Read and write traffic

Complicated Example: Putting it all Together


40/98



Complicated Example: Putting it all Together

Each App Virtual Machine reads to & writes from same NAS

Something caused excessive writes from each Virtual Machine

Increased CPU usage per Virtual Machine

Increased write traffic per Virtual Machine

Ton of writes onNAS VM!

Issue

Bug in application within Virtual Machine caused error condition

Error condition caused excessive writes to same NAS

- Network traffic for application VMs, disk traffic on NAS VM

Each Virtual Machine is so busy writing that it never reads



41/98




Use Cases: What do look for and why

CPU

Memory

Disk

Network

Network Performance Problems 101


42/98



Network Performance Problems 101

What should I look for to figure out if network is an issue?

Am I getting the packet rate that I expect?

Am I getting the bandwidth (read/write) I expect?

Is all traffic on one NIC, or spread across many NICs?

[more advancednot available through counters]: out-of-order packets?

What are some things I can do?

Check host networking settings (full-duplex/half-duplex, 10Gig network vs

100Mb network?, firewall settings)

Check VM settings: all VMs on proper networks?

Useful networking metrics


43/98



Useful networking metrics

Metric (Client) Metric(esxtop)


Packets

transmitted (insampling interval)

PKTTX/s net.transmitted.average Packets

transmitted insampling interval

Packets received(in sampling

interval)

PKTRX/s net.received.average Packets receivedin the sampling

interval

Data transmitrate (KBps)

MbTX/s net.transmitted.average Amount of datatransmitted per

second*

Data receive rate

(KBps)

MbRX/s net.received.average Amount of data

received persecond*


Network Performance Troubleshooting


44/98



Network Performance Troubleshooting

Customer complains about slow network

Shes running netperf on a GigE Link

She sees only 200Mbps

Why? I bet its that VMware stuff!!

- Note to reader: Please dont blame VMware first

Where do we start?

Where do we begin? Check VM Bandwidth.


45/98



b g

Measure VM Bandwidth (net.transmitted.average)

200 Mb/s

Screenshot from the vSphere client

Check Host Bandwidth


46/98

46 Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMwareproducts are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. inthe United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.


Measure Host Bandwidth (net.transmitted.average)

Host sees around 900Mbpswhy is VM at 200Mbps?

Hmmare we sharing this NIC with multiple VMs?

All VMs using same NIC (VM Network)


47/98



All VMs using VM Network and sharing 1 physical NIC

All Traffic is Going Through One Nic!


48/98



Measure per-physical-NIC traffic

Hmmall VM traffic is going through 1 NIC

Lets split the VMs across NICs

All traffic through one

NIC on this host

Split VMs Across Multiple NICs. Bingo!


49/98



So Complicated! Where do I begin?


50/98



So much to track

But I have different needs! For example

What if I want esxtop for multiple VMs at once?


51/98



vTop: A stats display similar to esxtop, but showing multiple metrics per VM in onescreen

What if I want to see if DRS is working?


52/98





Highlight the effectiveness of DRS by overlaying stats and events data

What if I Want to Compare Guest/Host Metrics on One Chart?


53/98





Overlay stats data from the ESX and the GOS

What If I Want To Compare the Throughput of 2 RPs?


54/98





Aggregate stats data by resource pools, clusters, group of VMs(Windows, Linux),vApps, etc.

Sounds Greatbut how do I do this stuff?


55/98





Basic Steps

1. Collect the Stats you Need

2. Interpret the Stats

3. Present the stats

Observations

1. Stats collection is similar across customer environments

2. Interpretation/Presentation is specific to customer

Stats collection is a major pain point

1. For customers: so many stats, complex API

2. For us: we get the same questions over and over again

Soweve created a reference application for Stats Collection

Agenda


56/98





Use Cases


Q & A



57/98





Monitoring Applications consist of a few standard parts:

Selecting entities to monitor (hosts, VMs, etc.)

Selecting metrics to monitor (CPU, Memory, etc.)

Choosing collection frequency

Keeping abreast of inventory changes

Collecting stats

Presenting data in a meaningful way

StatsFeeder: A Reference Monitoring Application


58/98





Present data in a meaningful way

Collect statistics(note: many sources!)

Keep track of inventory changes

Select Entities andMetrics to monitorand set collection

frequency

A Brief Walkthrough of StatsFeeder


59/98





1. Select Entities/Metrics to Monitor

2. Keep Track of Inventory Changes

3. Collect Statistics

4. Present Statistics

Selecting Entities/Metrics to Monitor


60/98





Relevant Modules in StatsFeeder

1. Counter Info Initializer

2. User Input Reader

3. Entity MoRef Retriever

4. Entity Metrics and Performance Provider Summary

Counter Info Initializer


61/98





Retrieves available counters.

Tip:

Counter information is static:

retrieve only once.

Counter Info Initializer: Code Snippet


62/98





The PerformanceManager managed object contains a property with

the name perfCounter

It is an array of type PerfCounterInfo.

Use PropertyCollector to retrieve this property and get the

supported counters.

PropertySpec pSpec = new PropertySpec();

pSpec.setType(PerformanceManager);

pSpec.setPathSet(new String[]{perfCounter});

ObjectSpec oSpec = new ObjectSpec();

oSpec.setObj(serviceContent.getPerfManager());

User Input Reader


63/98





Reads the configuration file for:

The entities for which the

performance statistics are being

retrieved.

Specify individual entities by name

Specify the name of the container andthe child type to monitor the entities in

that container

The performance statistics to

retrieve. The frequency at which the

performance data should be

retrieved.

User Input Reader: Sample Configuration File


64/98

64





Entity MoRef Retriever

R i M R f f i i


65/98

65





Retrieves MoRefs of entries in

configuration file.

Tricks:

MoRef does not change once an item is

added to the vCenter inventory.

MoRef does change if entity is removed

and re-added.

Therefore:

Maintain cache of MoRefs Monitor changes to the inventory

Use Views to retrieve MoRefs

Entity MoRef Retriever

R t i th M R f f i di id l titi i P t C ll t


66/98

66





Retrieve the MoRef of individual entities using PropertyCollector or

SearchIndex.

Retrieve the MoRefs of entities in a container using ContainerView:

Obtain the MoRef of the container

Create the ContainerView ContainerView has a property with the name view and it is an array of type

ManagedObjectReference. This contains the MoRefs of all the types specified

during the ContainerView creation.

vimPort.createContainerView( serviceContent.getViewMgr,

containerMoRef,

new String[]{VirtualMachine},//MO typetrue //recursively look in the container

);

Entity Metrics and Performance Provider Summary

R t i th m t i ID f ll th


67/98

67





Retrieve the metric IDs for all the

entities.

E.g.: For dual CPU VM, metric IDs are 0

and 1 for CPU 0 and CPU 1.

Caveat: Cant get this for powered off

VMs and host systems, disconnectedhosts and orphaned VMs

Tricks:

Performance Metric IDs for entities do

not changeif hardware does not

change.

Therefore:

Cache Metric IDs for entities.

Tip:

Use wild card for metric IDs

Entity Metrics


68/98

68





vimPort.queryAvailablePerfMetric (

serviceContent.getPerfManager(),

entityMoRef,

null,

null,

intervalID //If not specified, defaults to

//available metrics for historical stats.

);

Entity Metrics and Performance Provider Summary

Performance Provider Summary


69/98

69






provides information about:

The type of statistics it supports (real-

time or historical)

The refresh rate for statistics

generation

Performance Provider Summary datais the same for managed objects ofthe same type.

Tricks:

You dont have to retrieve this data for

all entities.

Make sure the interval between twoperformance data queries is equal to at

least the refresh rate to avoid getting

the same value.



70/98

70

Cop

Documents

performancemonitoring-100409184324-phpapp01