SER1289BE vSphere Troubleshooting Tips and … 1 Use of the VMware vSphere Command-line Interface (vCLI), the ESXi Shell, and Log Files to Diagnose and Correct Problems 2 Troubleshooting

Javier Menendez & Francis Daly

SER1289BE

#VMworld #SER1289BE

vSphere Troubleshooting Tips and Tricks

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#SER1289BE CONFIDENTIAL 2



bution

Agenda

1Use of the VMware vSphere Command-line Interface (vCLI),

the ESXi Shell, and Log Files to Diagnose and Correct Problems

2 Troubleshooting Networking Issues

3 Troubleshooting Storage Issues

4 Troubleshooting vSphere HA Cluster and Performance Issues

3#SER1289BE CONFIDENTIAL



bution

4Certain troubleshooting topics are addressed in the VMware vSphere: Fast Track [6.5] course.

Scope

This Presentation covers portions of the Troubleshooting Workshop:

For more information, go to http://vmware.com/education



bution

Troubleshooting Overview



bution

Troubleshooting Process

The troubleshooting process consists of the following tasks:

6

Defining the problem

• Identifying symptoms

• Gathering information

Identifying the cause of the problem

• Identifying possible causes

• Determining the root cause

Solving the problem

• Identifying possible solutions

• Implementing the best solution

#SER1289BE CONFIDENTIAL



bution

Solving the problem … Can I do it with the Web Client or should I use the esxi shell?

7

What are my choices:

• Can I fix the problem with the Web, Host or vSphere Client?

• Do I want or need to use the command line utilities?




bution

Troubleshooting Tools and Choices



bution

Command-line Troubleshooting Choices

Choices to obtain command-line access on a VMware® ESXi™ host:

9

vSphere Management Assistant (VMA)

VMware vSphere® ESXi™ Shell

• Can be access via:

• 1. The DCUI (if enabled via ALT + F1)

• 2. Via putty after enabling ssh.

• Appliance deployed via ovf

• Contains vCLI package

• Does NOT use port 22.




bution

The esxcli Command

The esxcli command

offers the following

namespaces,

as well as many new esxcli functions added in

the vSphere 6.5.

If working with vSAN,

check out the new esxcli

vsan debug namespace




bution

Example: The esxcli vsan Namespace

The vSAN related

commands are found in

esxcli vsan




bution

New commands added with every new vSphere release.

Ex: The esxcli vsan debug Namespace

The new vSAN debug

related commands are

amazing to discover the

state of your objects, vms,

etc.




bution

esxcli esxcli command list

displays all the command

options.

To quickly find the command

needed, combine it with the

grep command

Tip: Remember the esxcli esxcli command list command




bution

The VMA’s vicfg-* Commands

14

vicfg- [tab] [tab]

• Commands with the vicfg- prefix enable you to manage

your storage, network, and host configuration.

• Type vicfg- followed by the TAB key twice so see all the available commands.

• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.




bution

The esxi shell’s esxcfg- Commands are still there

15

esxcfg- [tab] [tab]

• Commands with the esxcfg-

prefix enable you to manage

your storage, network, and

host configuration.

• Type esxcfg- followed by the TAB key twice so see all the available commands.

• Remember the –h (help) option to master these commands.




bution

Use the –h option (for help)

to see options for this

particular command and in

some cases, even

examples on how to use the

command.

How many times did you

struggle to create a

vmkernel port with the cli?

Tip: Remember the “-h” option for esxcfg-* commands




bution

Manipulating virtual machines with the cli from the V.M.A.?

17

• vmware-cmd

– The vmware-cmd command is exclusively used for virtual machines.

vmware-cmd

• The vmware-cmd command is used to interact with virtual machines.




bution

Example: Listing your Virtual Machines

18

vmware-cmd -l

• Lists the virtual machines that are located on the target host. Lists virtual machines by path to the .vmx file.




bution

Tip: Learn More On Your Own

19

Type vmware-cmd --help to

see what the command allows

you to do.

vmware-cmd allows you to

start a vm, take snapshots,

register a vm, etc.

The esxi host has a similar

command to manipulate vms

called vim-cmd vmsvc.




bution

Manipulating virtual machines from the esxi shell

20

vim-cmd vmsvc can

perform similar

tasks from the

esxi shell.

Wondering what

the destroy

option does?

Better try it on

a test vm.




bution

Tip: Location of VMware vCenter Server 6.5 Log Files

21

The VMware vCenter Server

6.5 logs are located in the

%ALLUSERSPROFILE%\VM

Ware\vCenterServer\logs

folder.

The VMware vCenter Server

Appliance 6.5 logs are

located in the

/var/log/vmware folder.

Recommendation

See http://kb.vmware.com/kb/2110014 and 1021806 for additional details.




bution

Useful ESXi Host Logs for Troubleshooting

ESXi hosts write to multiple log files, depending on which action is being performed.

22

Log file Purpose

hostd.log Host Management service logs

syslog.logManagement service initialization, watchdogs, scheduled

tasks, and DCUI use

vmkernel.log

Core VMkernel logs, including device discovery, storage

and networking device and driver events, and virtual

machine startups

vmkwarning.logA summary of warning and alert log messages excerpted

from the VMKernel logs

vmksummary.log

A summary of ESXi host startup and shutdown, and an

hourly heartbeat with uptime, number of virtual machines

running, and service resource consumption




bution

Tip: Other Critical Logs to Be Aware Of and how to view them

23

/var/log/fdm.log displays HA

cluster related events and

notifications.

/var/log/shell.log displays

commands executed inside

of the esxi shell.

/var/log/clomd.log displays vSAN

health related information for

devices among the output




bution

Tip: Remember /var/log

24

Lots of logs available in

this directory




bution

Collecting Diagnostic Data for

VMware Technical Support

25

Methods for collecting diagnostic information to send to VMware Technical Support include the following:

• Use the GUI to export files to a log bundle.

− vSphere Client or vSphere Web Client

• Use the vm-support command to collect

information from an individual ESXi host.




bution

Tip: Running vm-support when engaging support




bution

Network Troubleshooting Commands



bution

Tip: Useful Networking Related Commands

28

Use esxcfg-vmknic –l or

vicfg-vmknic -l to view your

vmkernel port related

information. Are they still there

and enabled?

Use esxcfg-vswitch –l or

vicfg-vswitch –l to view virtual

switch related information. Is

the switch connected to at

least one uplink?

Use esxcfg-nics –l or vicfg-

nics –l to view uplink related

information. Are the nics up or

down?

esxcli also has options to

perform these tasks.

For example:

esxcli network nic list is similar

to esxcfg-nics -l




bution

Somebody could accidentally

down a nic?

If so, type esxcli network nic up

–n vmnic2.

Tip: Did You Know That…




bution

Somebody could disable a vmkernel port

with commands like esxcfg-vmknic -D?

If so, type esxcfg-vmknic -e to re-

enable the vmkernel port.





bution

Somebody could accidentally

disconnect an uplink from a

virtual switch?

ffaaa

Tip: Rebuilding the Management Network from scratch




bution

There is a built-in packet

sniffer called pktcap-uw

and an older one called

tcpdump-uw.

Need a packet sniffer? Remember pktcap-uw




bution

Storage Troubleshooting



bution

You can see how many dastastores exist and how full

the datastores are? This command displays vmfs, nfs,

vsan and vvol datastores. There is also a vdf –h

command.





bution

You can easily view all

sorts of information

with esxcli?





bution

Use this command to

retrieve data about a

specific SSD device.

Critical for vSAN,Flash

Read Cache or any

datastore using SSDs.

vSphere 6.5

Includes

Tip: Use esxcli Storage Core Device Smart for SSD information

36 36



bution

Tip: Be familiar with partedUtil

37

partedUtil is the command

that partitions luns. It can also

create, resize and delete

partitions.



bution

Tip: Suspect metadata corruption?

38

voma is the on-disk metadata

analyzer for vmfs datastores.




bution

Troubleshooting Clusters and Performance



bution

vSphere HA

40

vCenter

Server

ESXi host (slave)

FDM

ESXi host (master)

FDM

ESXi host (slave)

FDM

vpxd

hostdhostdhostd

Management

Network

vpxa vpxa vpxa

Heartbeat

Datastores

A reliable network connection between the hosts and VMware® vCenter Server™

is essential for enabling vSphere HA.



bution

There is a log for HA that

you can’t see with the Web

or vSphere Client?

Tip: Remember F.D.M




bution

You can stop, start or restart the HA service from the cli if needed?





bution

Tip: You can restart all services at once if needed …

43

Remember services.sh restart




bution


44

esxtop is an amazing tool for

performance troubleshooting?

esxtop has options for CPU

(c), RAM (m), Disk (d,u,v)

and Network (n) related

statistics?

esxtop has an option for vSAN

(-x option)?




bution

esxtop has built-in help?





bution

You can add more

output to esxtop by

pressing “f”?

For example, you can

add NUMA related

information (G Option)





bution

Tip: Learn the vsish command and its capabilities

47

vsish has hundreds of

subcommands.

Just type the command and

take a look …

You can type commands like

cd to change directories and

get to view information.




bution

Tip: Learn the vsish command and its capabilities

vsish has hundreds of

subcommands.

Just type the command and

take a look …




bution

Additional Education Resources

At VMworld 2017 Europe

• Education & Certification Lounge: VMvillage, Hall 6

• Certification Exam Center: CC5

Online

• VMware Training: www.vmware.com/education

• VMware Certification: www.vmware.com/certification

Global Support Services

Learn more about how VMware is radically transforming Customer Support through VMware Skyline™ technology.

• Visit the demo booth within the Solutions Exchange

• Sign up for a Meet the Experts session in the Content Catalogue

• Visit www.vmware.com/support/service/skyline

Save 50% off VCP & VCAP

exams at

VMworld 2017




bution

http://www.vmware.com/education

http://www.vmware.com/certification

http://www.vmware.com/support/service/skyline

Prove Your Expertise with VMware Digital Badges

What are VMware Digital Badges?

• Single source that combines your credential and

a complete overview of your skills

• Way for you to easily share your accomplishments

in social media

• Provides employers with easy, valid verification of

VMware credentials

Digital Badges Online

• vmware.com/go/vSAN2017badge

• vmware.com/go/VxRail2017badge

VMware vSAN 2017 Specialist

Dell EMC VMwareCo-Skilled - VxRail 2017

NEW!VMware Digital Badges

Prove your expertiseon vSAN and

vSAN HCI environments




bution

http://www.vmware.com/go/vSAN2017badge

http://www.vmware.com/go/VxRail2017badge



bution



bution

Documents

SER1289BE vSphere Troubleshooting Tips and … 1 Use of the VMware vSphere Command-line Interface (vCLI), the ESXi Shell, and Log Files to Diagnose and Correct Problems 2 Troubleshooting