Citrix XenServer 5.5 Troubleshooting

  • View
    23.365

  • Download
    4

Embed Size (px)

DESCRIPTION

Citrix XenServer 5.5 Troubleshooting Deep Dive.

Text of Citrix XenServer 5.5 Troubleshooting

  • 1. Citrix XenServer
    • Webcast 26. Mrz 2010

Advanced Troubleshooting Thomas Krampe Citrix Technology Professional 2. Agenda 04/19/10 (c) 2009 Thomas Krampe

  • Architecture
  • Troubleshooting
  • Birth and Death of a VM
  • Object modell & CLI
  • Unattended Setup & Configuration
  • Question & Answers

3. Architecture 04/19/10 (c) 2009 Thomas Krampe

  • Architecture
  • XenServer Architecture
  • Xen Domain0
  • Guest OS Internals
  • XAPI
  • Storage Manager
  • CLI
  • GUI (XenCenter)

4. XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI NFS iSCSI FC 5. Xen Domain0 04/19/10 (c) 2009 Thomas Krampe

  • Dom0 kernel and initrd passed on start up
  • Dom0 has direct access to HW
  • Bootloader
  • /boot/grub/grub.conf
  • /boot/extlinux.conf
  • [Ctrl-A]3 times will trap into the Xen debug console

6. Guest OS internals 04/19/10 (c) 2009 Thomas Krampe PV Guest

  • Modified Kernel (Xenified)
  • Kernel name will have xs somewhere in the filename
  • Guest agent reports VM stats
  • Use pygrub bootloader

HVM Guest (Windows)

  • Unmodified Kernel
  • Need PV drivers
  • Event Channel, Network and Block
  • Guest agent uses WMI

7. XAPI 04/19/10 (c) 2009 Thomas Krampe

  • /opt/xensource/bin/xapi
  • Service, autostarted on boot
  • Implements the Xen API spec
  • Talks to Storage Manager (sm) for SR control operations
  • Stores configuration in sqlite DB
  • /var/xapi/state.db
  • sqlite3/var/xapi/state.db
  • DB is replicated on all hosts in a pool
  • Sets up SR defaults on firstboot
  • Failing firstboot scripts might cause issues
  • /var/xapi/generate-*and/var/xapi/firstboot-*
  • Defaults from/etc/xensource-inventory

8. Storage Manager 04/19/10 (c) 2009 Thomas Krampe

  • Manages SR control operations to various backends
  • /opt/xensource/sm/sm.py
  • Plugin model
  • One plugin for each SR type
  • /opt/xensource/sm/SR.py
  • Error codes listed in/opt/xensource/sm/XE_SR_ERRORCODES.xml
  • xe sm-listlists all supported drivers

9. CLI 04/19/10 (c) 2009 Thomas Krampe

  • Local or Remote
  • Talks to XAPI
  • Bash auto completion
  • xe.exe (for Windows)
  • xe helporxe help --all

10. GUI 04/19/10 (c) 2009 Thomas Krampe

  • Uses Microsoft .NET framework
  • Uses https/xml-rpc to communicate with XAPI
  • VNC Client for guest frame buffer
  • VNC traffic is tunneled over https
  • RDP client for Windows guests

11. Troubleshooting 04/19/10 (c) 2009 Thomas Krampe

  • Troubleshooting
  • Server Install
  • Useful Linux Commands
  • Server Diagnostics & Inventory
  • Saving System State
  • XS Server Log Files
  • XenSource Agent
  • XenCenter Logging
  • Summery

12. Server Install 04/19/10 (c) 2009 Thomas Krampe

  • After Product CD Boots
  • Alt-F2displays Linux login prompt
    • Login: root (no password required)
  • Alt-F3displays install event log
  • Alt-F1displays original install menu
  • Support Tarball
  • support.sh (creates /tmp/support.tar.bz2)
  • Support tarball is created in /tmp on install error
  • Support tarball is saved in /root after successful install

13. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe

  • Storage
  • List Disks detected
    • fdisk l
  • Network
  • Show NIC configuration
    • ifconfig eth0 [eth1 ..]
    • ifup eth0
  • TCP/IP Tools
    • ping, route -n
    • dhclient eth0

14. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe

  • Process Management
  • ps Af
  • top
  • xentop

15. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

  • Server Diagnostics
  • xen-bugtool
  • Run on the XS Console
  • Records all log files
  • Writes compressed file for submission to L2 support
  • /var/opt/xen/bug-report/bug-report-xxx.tar.bz2

16. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

  • Server Diagnostics
  • Automatic Upload for Support
  • xe host-bugreport-upload
  • Can be run from admin system
  • Runs xen-bugtool and uploads to XS Support
  • Note unique ID of report for support

17. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe

  • Server Inventory
  • system-info.sh | less
  • Interactive script that captures all non-log state
  • Hardware devices & Device drivers
  • Storage repository and VM configurations

18. Saving System State 04/19/10 (c) 2009 Thomas Krampe

  • xen-bugtoolarchives state and configuration files
  • Network config
  • Install options
  • Install logs
  • Storage Repository metadata
  • VM configs
  • Xenstore

19. XS Server Log Files 04/19/10 (c) 2009 Thomas Krampe

  • XenSource XAPI Agent
  • /var/log/messages
  • Contains other system logging
  • /var/log/xensource.log
  • Xen/Domain 0
  • Xen Boot Messages
  • xe host-dmesg
  • Domain 0 Boot Messages
  • dmesg

20. Debug Logging for XAPI Agent 04/19/10 (c) 2009 Thomas Krampe

  • Enable debug logging for all subsystems
  • xe log-set-output level=debug output=file:/tmp/debug-all
  • Debug for only a certain subsystem
  • xe log-set-output level=debug key=storage output=file:/tmp/storage-debug
  • List Available Subsystems
  • xe log-get-keys
  • Restore normal logging
  • xe log-set-output level=debug output=nil
  • xe log-set-output level=debug key=storage output=nil

21. XenSource Agent 04/19/10 (c) 2009 Thomas Krampe

  • XS Agent runs on top Xen Hypervisor API and XenStore
  • Can be restarted without harming VMs or other members of the pool
  • service xapi restart
  • If the UI seems to get stuck or goes into a weird state, restart the agent and reconnect the UI
  • Another simular command isxe-toolstack-restart

22. XenCenter Logging 04/19/10 (c) 2009 Thomas Krampe

  • Location
  • {user home}Application DataXenSourceXenCenterlogsXenCenter.log
  • Rolling
  • Log files will get rolled
  • XenCenter.log.#
  • Increase Log Level
  • C:Program FilesXenSourceXenCenter directory

23. Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Support Representative. xe host-bugreport-upload runs xen-bugtool and uploads the result to XenSource support. xe host-bugtool can be used to save Domain 0 state, to assist during emergency recovery Use Linux commands (network, storage, modules) to troubleshoot server install. 24. Birth and Death of a VM 04/19/10 (c) 2009 Thomas Krampe

  • Birth and Death of a VM
  • VM Installation Internals
  • Behind the Scenes
  • Debugging VMs

25. VM Installation Internals 04/19/10 (c) 2009 Thomas Krampe

  • Debian Sarge/Etch
  • Root file system is extracted into the VDI
  • Debian Xen kernel
  • RHEL (3.x,4.x,5) / SLES (9.x, 10.x)
  • Uses regular repositories
  • RHEL / SLES Xen kernel
  • Windows (all)
  • Installs from CD/ISO bits
  • Unmodified Windows kernel

26. Behind the Scenes 04/19/10 (c) 2009 Thomas Krampe

  • Paravirtualized VMs use pygrub
  • Simple python based bootloader for Xen
  • Finds kernel and initrd from VMs boot device
  • CPU then jumps to this kernel to start this VM
  • HVM VMs use hvmloader
  • hvmloader creates a