Citrix XenServer 5.5 Troubleshooting

• 1. Citrix XenServer
  • Webcast 26. Mrz 2010

Advanced Troubleshooting Thomas Krampe Citrix Technology Professional 2. Agenda 04/19/10 (c) 2009 Thomas Krampe

• Architecture

• Troubleshooting

• Birth and Death of a VM

• Object modell & CLI

• Unattended Setup & Configuration

• Question & Answers

3. Architecture 04/19/10 (c) 2009 Thomas Krampe

• Architecture

• XenServer Architecture

• Xen Domain0

• Guest OS Internals

• XAPI

• Storage Manager

• CLI

• GUI (XenCenter)

4. XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI NFS iSCSI FC 5. Xen Domain0 04/19/10 (c) 2009 Thomas Krampe

• Dom0 kernel and initrd passed on start up

• Dom0 has direct access to HW

• Bootloader

• /boot/grub/grub.conf

• /boot/extlinux.conf

• [Ctrl-A]3 times will trap into the Xen debug console

6. Guest OS internals 04/19/10 (c) 2009 Thomas Krampe PV Guest

• Modified Kernel (Xenified)

• Kernel name will have xs somewhere in the filename

• Guest agent reports VM stats

• Use pygrub bootloader

HVM Guest (Windows)

• Unmodified Kernel

• Need PV drivers

• Event Channel, Network and Block

• Guest agent uses WMI

7. XAPI 04/19/10 (c) 2009 Thomas Krampe

• /opt/xensource/bin/xapi

• Service, autostarted on boot

• Implements the Xen API spec

• Talks to Storage Manager (sm) for SR control operations

• Stores configuration in sqlite DB

• /var/xapi/state.db

• sqlite3/var/xapi/state.db

• DB is replicated on all hosts in a pool

• Sets up SR defaults on firstboot

• Failing firstboot scripts might cause issues

• /var/xapi/generate-*and/var/xapi/firstboot-*

• Defaults from/etc/xensource-inventory

8. Storage Manager 04/19/10 (c) 2009 Thomas Krampe

• Manages SR control operations to various backends

• /opt/xensource/sm/sm.py

• Plugin model

• One plugin for each SR type

• /opt/xensource/sm/SR.py

• Error codes listed in/opt/xensource/sm/XE_SR_ERRORCODES.xml

• xe sm-listlists all supported drivers

9. CLI 04/19/10 (c) 2009 Thomas Krampe

• Local or Remote

• Talks to XAPI

• Bash auto completion

• xe.exe (for Windows)

• xe helporxe help --all

10. GUI 04/19/10 (c) 2009 Thomas Krampe

• Uses Microsoft .NET framework

• Uses https/xml-rpc to communicate with XAPI

• VNC Client for guest frame buffer

• VNC traffic is tunneled over https

• RDP client for Windows guests

11. Troubleshooting 04/19/10 (c) 2009 Thomas Krampe

• Troubleshooting

• Server Install

• Useful Linux Commands

• Server Diagnostics & Inventory

• Saving System State

• XS Server Log Files

• XenSource Agent

• XenCenter Logging

• Summery

12. Server Install 04/19/10 (c) 2009 Thomas Krampe

• After Product CD Boots

• Alt-F2displays Linux login prompt

• • Login: root (no password required)

• Alt-F3displays install event log

• Alt-F1displays original install menu

• Support Tarball

• support.sh (creates /tmp/support.tar.bz2)

• Support tarball is created in /tmp on install error

• Support tarball is saved in /root after successful install

13. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe

• Storage

• List Disks detected

• • fdisk l

• Network

• Show NIC configuration

• • ifconfig eth0 [eth1 ..]

• • ifup eth0

• TCP/IP Tools

• • ping, route -n

• • dhclient eth0

14. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe

• Process Management

• ps Af

• top

• xentop

15. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

• Server Diagnostics

• xen-bugtool

• Run on the XS Console

• Records all log files

• Writes compressed file for submission to L2 support

• /var/opt/xen/bug-report/bug-report-xxx.tar.bz2

16. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

• Server Diagnostics

• Automatic Upload for Support

• xe host-bugreport-upload

• Can be run from admin system

• Runs xen-bugtool and uploads to XS Support

• Note unique ID of report for support

17. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

• Server Inventory

• system-info.sh | less

• Interactive script that captures all non-log state

• Hardware devices & Device drivers

• Storage repository and VM configurations

18. Saving System State 04/19/10 (c) 2009 Thomas Krampe

• xen-bugtoolarchives state and configuration files

• Network config

• Install options

• Install logs

• Storage Repository metadata

• VM configs

• Xenstore

19. XS Server Log Files 04/19/10 (c) 2009 Thomas Krampe

• XenSource XAPI Agent

• /var/log/messages

• Contains other system logging

• /var/log/xensource.log

• Xen/Domain 0

• Xen Boot Messages

• xe host-dmesg

• Domain 0 Boot Messages

• dmesg

20. Debug Logging for XAPI Agent 04/19/10 (c) 2009 Thomas Krampe

• Enable debug logging for all subsystems

• xe log-set-output level=debug output=file:/tmp/debug-all

• Debug for only a certain subsystem

• xe log-set-output level=debug key=storage output=file:/tmp/storage-debug

• List Available Subsystems

• xe log-get-keys

• Restore normal logging

• xe log-set-output level=debug output=nil

• xe log-set-output level=debug key=storage output=nil

21. XenSource Agent 04/19/10 (c) 2009 Thomas Krampe

• XS Agent runs on top Xen Hypervisor API and XenStore

• Can be restarted without harming VMs or other members of the pool

• service xapi restart

• If the UI seems to get stuck or goes into a weird state, restart the agent and reconnect the UI

• Another simular command isxe-toolstack-restart

22. XenCenter Logging 04/19/10 (c) 2009 Thomas Krampe

• Location

• {user home}Application DataXenSourceXenCenterlogsXenCenter.log

• Rolling

• Log files will get rolled

• XenCenter.log.#

• Increase Log Level

• C:Program FilesXenSourceXenCenter directory

23. Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Support Representative. xe host-bugreport-upload runs xen-bugtool and uploads the result to XenSource support. xe host-bugtool can be used to save Domain 0 state, to assist during emergency recovery Use Linux commands (network, storage, modules) to troubleshoot server install. 24. Birth and Death of a VM 04/19/10 (c) 2009 Thomas Krampe

• Birth and Death of a VM

• VM Installation Internals

• Behind the Scenes

• Debugging VMs

25. VM Installation Internals 04/19/10 (c) 2009 Thomas Krampe

• Debian Sarge/Etch

• Root file system is extracted into the VDI

• Debian Xen kernel

• RHEL (3.x,4.x,5) / SLES (9.x, 10.x)

• Uses regular repositories

• RHEL / SLES Xen kernel

• Windows (all)

• Installs from CD/ISO bits

• Unmodified Windows kernel

26. Behind the Scenes 04/19/10 (c) 2009 Thomas Krampe

• Paravirtualized VMs use pygrub

• Simple python based bootloader for Xen

• Finds kernel and initrd from VMs boot device

• CPU then jumps to this kernel to start this VM

• HVM VMs use hvmloader

• hvmloader creates a