SUM303: XenDesktop Advanced Troubleshooting

Mick Glover

Senior Support Readiness Specialist

October 16, 2012

Agenda

• Overview

• XD5.x Architecture Basics

• Logging & Tracing

• Key Support Tools Overview

• Using PowerShell to monitor your site

• Troubleshooting Example(s)

Overview

Before we begin..A Question

Q. To date, what is the most common problem logged with technical support

worldwide in 2012?

A. VDA Connectivity issues (Registration, Connection/Disconnection…)

The Good News…

• Its not all about verbose Logging and Tracing…

• The following goes a long way…○ Good Understanding of Product Architecture○ Knowing what to expect○ Knowing what to trace and when

XD5.x Architecture Basics

Machine CreationServices

BrokerService

InfrastructureServices

XenDesktop 5 Architecture - .Net SOA

Broker

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Desktop Studio Desktop Director

WCF [80]

Virtual Desktop

Agent (VDA)

WinRM 2.0

PVS for VMs Agent

[5985/5986]

WCF [80]

PoSH PoSH

Group Policy Engine

SQL Server SQL Server SQL Server7

ServicesKey Responsibilities

• Broker Service

○ VDA registration, Resource allocation, Power MGMT & Licensing enforcement

• Configuration Service

○ Provides Directory Services, MetaData storage & security

• Host Service

○ Manages Hypervisor connection & resources

ServicesKey Responsibilities

• Machine Creation Service

○ Creates & manages VMs and Master images

• Machine Identity Service

○ Creates & manages the disks attached to VMs

• AD Identity Service

○ Creates & manages AD Computer a/c’s

Registration Basics

• VDA registration○ Process of locating a controller and establishing ongoing communication○ WCF web service endpoints on both ends facilitate the registration process

• Registration methods○ Registry base registration○ AD base registration○ Quick Deployment registration

• Registration types ○ Soft Registration○ Hard Registration

WCF Registration portSpecifics

• XD5.x: TCP/IP port 80

○ Controller and VDA

• Ports can be set differently on controller and VDA

• Special consideration when changing the WCF port on controllers if using

Registry based registration i.e. ControllerRegistrarPort

Configuring Registry based registrationListOfDDCs

Checking the WCF port on a ControllerBrokerService.exe /show

Changing the WCF port on a ControllerBrokerService.exe –vdaport 89

Changing the WCF ports on the VDA

• Re-configure the VDA ○ (add/remove programs)

• HDX Computer Policy

Changing the WCF ports on the VDAConfirming the changes (registry)

Changing the WCF ports on the VDAConfirming the changes (.Config file)

ControllerRegistrarPortDetails

• Hardcoded on VDA to use port 80

• Value must be manually changed if VDA

WCF port is changed on Controller

• When using AD based registration

○ WCF port is included as part of the URL

attribute and therefore this manual change is

not required as each controller will update its

own details in AD

ControllerRegistrarPortRegistry value created after re-configuring the VDA

ControllerRegistrarPort - Known IssueChanging Virtual Desktop Agent Port Using Add/Remove Programs Fails

• Cause: ControllerRegistrarPort is created

incorrectly with an ending space

• Resolution: Manually delete and recreate

Registry value

• Note: CTX131263 also includes updated

binary's which can be deployed to resolve

the issue without having to manually edit

the registry

VDA HeartbeatDetails

• Value: HeartbeatPeriodMs

○ Originally a 1 minute timeout (60000ms) with the VDA sending

a ping every 30 seconds (an interval of half the timeout value)

• Since the release of XD5 SP1, the default

timeout/ping interval has changed to 10 & 5 minutes

respectively…

Logging & Tracing

.NET <Service>.exe.Config filesAvailable logging mechanisms

• "LogToCDF" - if present and set to 1, will log to CDF using the supplied CDF

trace module

• "LogFileName" - if present, will log to the specified file name

○ "OverwriteLogFile" - if present, will overwrite existing log to the file if it already exists

• Note: More than one sink can be specified, in which case logs will be output to

all of them…

Out of the box…

• CDF Tracing enabled for all services

• <Service>.exe.Config <appSettings>

</appSettings>

Out of the box…

• CDF Tracing enabled for all services

• <Service>.exe.Config <appSettings>

</appSettings>

• Clear text logging disabled

WCF Service Logging - ExampleEnabling clear text logging

1. Edit WorkStationAgent.exe.Config

2. Add the lines in Green below to enable and control clear text logging

3. Set appropriate NTFS permissions

4. Restart Citrix Desktop Service (WorkstationAgent.exe)

Service configuration logging reference - XD4

Service configuration logging reference - XD5

PortICA logging

• PortICA can produce both CDF and

clear text output for debugging

purposes○ Disabled by default

• Create PorticaConfig.XML and place

under the following directory on the

VM○ C:\Program Files\citrix\ICAService\XML○ XML directory must be manually created Note: Once PorticaConfig.xml has been created and

placed in the correct directory, it must be configured29

Configuring PorticaConfig.XML

• LogFile○ Level 0,1,5,9 supported○ 0 = disabled

• CdfTrace○ Level 1,5,9 supported

• FunctionTrace○ Values 1,5,9 supported○ Master flow control

<?xml version="1.0" encoding="utf-8"?><Config xmlns="Portica.xsd"> <Portica> <LogFile> <LogLevel>5</LogLevel> </LogFile> <CdfTrace> <LogLevel>5</LogLevel> </CdfTrace> <FunctionTrace> <LogLevel>5</LogLevel> </FunctionTrace> </Portica></Config>

Note: Cdf tracing is enabled by default for PortICA. The above values just control the verbosity of the output.

PortICA clear text logs

• Logs files are automatically saved to

the following locations○ XP C:\Documents and Settings\

LocalService\Local Settings\Temp ○ Win7 C:\Windows\ServiceProfiles\

LocalService\AppData\Local\Temp

Making the process easier…Logging Enabler v2.0

Logging Enabler v2.0VDA

Don’t forget Windows Event ViewerApplication Log - VDA

•XenDesktop 5.5 - VDA Event Log Messages http://support.citrix.com/article/CTX133154

Don’t forget Windows Event ViewerApplication Log - Controller

•XenDesktop 4.0 - VDA and DDC Event Log Messages - http://support.citrix.com/article/CTX133152

Finding & Validating ControllersVDA.log (WorkstationAgent.exe)

Confirming Successful Soft RegistrationBroker.log (BrokerService.exe)

Tips for searching Logs con’t

Soft Registration

• Soft registration ○ VDAs are not yet belong to a desktop group○ Only partial registration action done○ No config sent (XML) or status monitoring initiated○ No session launching or management is permissible○ VDA appears in Desktop Studio as registered○ Avoids unwanted registration delays○ Avoids network traffic and event logs

Viewing Soft Registered MachinesXDDBDiag - XenDesktop Site Summary

Hard Registration

• Promotes soft to hard registration○ Once VDA is added to a group○ Configuration data send to site database○ Status monitoring initiated (ready or

power-off etc.)

CDF Tracing overview

• The acronym goes by several names○ Citrix Diagnostic Facility○ Common Diagnostic Facility○ Common Diagnostic Function

• CDF is a system to capture diagnostic output from specific modules in Citrix

products which can be very helpful when troubleshooting issues

• The trace messages are coded into the product and displayed with appropriate

status and error codes when run

CDF messages in the source code

Trace Message Format Files (TMFs)How it works

• Raw CDF events are captured as GUIDs from the Citrix modules. To convert

these to human-readable strings, the GUIDs need to be matched with their

corresponding descriptions.

• The mapping of GUIDs to descriptions is accomplished by the use of TMF files

which are similar to PDB debug files…

TMF FilesPublic & Private

•There are two varieties of TMF files: Public and Private○ Public TMF files reveal basic state and function information for customers and partners

○ Private TMFs are for internal Citrix use only and are used by escalation engineers and

development while debugging. These files contain references to source code.

•Public TMF server: http://ctxsym.citrix.com/tmfs/xaxd

• Public TMFs are also available at ftp.citrix.com for externally use

Key Support Tools

• Tracing & Logging○ CDFControl○ CDFCheck○ CDFMarker○ CDFMonitor○ LogEnabler

• Monitoring & Collection○ Scout○ XD Site Checker○ HDXMonitor

• Database and Services○ XDPing○ XDDBDiag

Tracing & Logging

CDFControl (1/2)

• The primary tool used to control and consume CDF traces

• All available Citrix modules

• Additional features include ○ performance data capture○ tracing on system startup○ common error code lookups○ expert control to zero-in on true errors and failures

• Available at http://support.citrix.com/article/CTX111961

CDFControl (2/2)

CDFControl remote tracing

• Start CDFControl

○ Select View -> Remote Providers

○ Verify no firewalls are in place

○ Verify remote registry service running

• Select modules

○ Tools -> Remote Tracing

○ Saved to sub directory of CDFControl

'CDFControl_Remote_Traces\%computername%'

Remote Providers Error

•Make sure you can connect to registry on remote machine…○ Ensure Remote Registry service is started on remote system○ Ensure firewall is not preventing communications

Multi-Domain/Non-Trust Workaround…

CDFControl startup tracing

• Start CDFControl – select module(s)○ Tools -> Options (to set trace file path)○ Tools -> Startup tracing -> Enable

○ ETL file located at Startup trace file path

CDFMarker v1.3

• Stand alone executable ○ CDFMarker.exe

• Injects a customizable CDF trace

statement into CDF trace to help locate

narrow search

• CDFMarker outputs its statements to

MF_Hook_MMHook and therefore this

module must always be selected

CDFCheckCDFChkCmd.exe

• Suspect a specific binary as root

cause of issue

CDFCheck

• BrokerService.exe○ CdsController○ CdsInterService○ ControllerDal

Monitoring & Collection

XenDesktop Site CheckerV1.1

• Features

○ Control Logging (Enable/Disable)

○ Restart Services

○ Check Service Instances

• Views

○ Site Details

○ Node Details

○ Active Site Services (leases)

○ Assignments (Desktop Groups)

ScoutV2.4.0.2

• Collects data points and CDF traces

• Data package(s) uploaded to Citrix

Technical Support

• Auto-Update

○ Help -> Check for updates

ScoutData points (1/2)

• Hardware

• Software

Computer Bios Memory Devices

Processor PNP device Logical drive Page file usage

Network adapter & configuration

Video controller Printers IDE controllerSCSI controller

OS/Service packs PNP driver Driver

Processes MS Hotfix Grop Policy Info (RSOP)

ScoutData points (2/2)

• Farm

Hypervisor Site Information Controllers Catalogs

Desktop groups Configured VDA Broker administrators Installed DB version

XDDBDiag output Broker hypervisor connection

Connection leg

ScoutConfiguration

ScoutCDF Tracing

ScoutCollecting & Uploading Data

ScoutCheck-Update

• Compares local version against

latest version online

• If the Local and Server versions

do not match, click Download…

Database and Services

XDPing Database & Services

• CLI based tool

○ Verify configuration settings on DDCs and VDAs

• Can be run on the local console or remotely

○ Xdping

○ xdping /host MyDDC01.citrix.com

XDDBDiag

• Performs consistency check against

DB and summarises findings…

• Outputs the following diagnostic

data into comma-separated value

(.csv) files

○ Site Information, Virtual Desktop Agent Information, Current Connections / Connection Log, Hypervisor Connections, Policy Information, Desktop Groups, Controller Information & SQL Information

XDDBDiag output

Using PowerShell to monitor your site

PowerShell - Reference

• Load XenDesktop Snap-ins

○ asnp citrix*

• Return all available XenDesktop cmdlets

○ Get-command –module citrix.*

• XenDesktop 5.6 SDK Cmdlet Help

○ http://support.citrix.com/article/CTX127254

PowerShell Remoting

• Built-in remoting functionality

• -AdminAddress Parameter

○ Specifies the host name or IP address of the controller to

which the PowerShell snap-in will connect

• Default = “LocalHost”

• Note: Once a value is specified by any command,

this value becomes the new default

PoSH - Validating Service Status

• Use Get-BrokerServiceStatus (Broker Service)

• Use Get-ConfigServiceStatus (Configuration Service)

• Use Get-HypServiceStatus (Host Service)

• Use Get-AcctServiceStatus (AD Identity Service)

• Use Get-ProvServiceStatus (Machine Creation Service)

• Use Get-PvsvmServiceStatus (Machine Identity Service)

Get-BrokerServiceStatus - Return Values (1 of 2)

Status Meaning

OK The broker is connected to a database that is valid, and the service is running.

DBUnconfigured The broker does not have a database connection configured

DBRejectedConnection The database rejected the logon from the Broker Service. This may be caused by bad credentials, or the database not being installed.

InvalidDBConfigured The database schema is missing (possibly just the stored procedures in it).

DBNotFound The specified database could not be located with the configured connection string.

DBMissingOptionalFeature The broker is connected to a database that is valid, but it does not have the full functionality required for optimal performance. Upgrading the database is advisable.

DBMissingMandatoryFeatureThe broker is connected to a database that is valid, but it does not have the full functionality required so the broker cannot function. Upgrading the database is required.

Get-BrokerServiceStatus - Return Values (2 of 2)

Status Meaning

DBNewerVersionThanService The broker is too old to use the database. A newer version is required.

DBOlderVersionThanService The database is too old for the Broker Service. Upgrade the database.

DBVersionChangeInProgress A database schema upgrade is in progress.

PendingFailure Connectivity between the Broker Service and the database has been lost. This may be a transitory network error, but may indicate a loss of connectivity that requires administrator intervention.

Failed Connectivity between the broker and the database has been lost for an extended period of time, or has failed due to a configuration problem. The broker service cannot operate while its connection to the database is unavailable.

Unknown The Service's status cannot be determined84

Setting the DB connection string(s)

• Required for troubleshooting, DB Migration…

• To view existing DB connection string○ Use Get-BrokerDBConnection

• Server○ Name of server hosting DB

• Database○ Name of XD5 database (CitrixXenDesktopDB = Default)

Setting the DB connection string(s) con’t

• Trusted_connection=true○ Enables Integrated Windows authentication

• The following error is returned if above parameter is not specified:

• Why? ○ Service instances use the local DDC machine a/c to authenticate to the DB○ SA authentication is not supported…

• To Reset Broker Service DB connection string○ Set-BrokerDBConnection –DBConnection $null

• The Broker Services will reset and return to an idle state until a valid new

database connection string is specified

• New value = DBUnconfigured

• To Set Broker Service DB connection string:○ Set-BrokerDBConnection –DBConnection

"Server=dbserver;Database=XDDB;Trusted_Connection=True“

• New value = OK

Setting the DB connection string(s) - ReferenceOne Cmdlet for each Service

• Set-BrokerDBConnection (Broker Service)

• Set-ConfigDBConnection (Configuration Service)

• Set-HypDBConnection (Host Service)

• Set-AcctDBConnection (AD Identity Service)

• Set-ProvDBConnection (Machine Creation Service)

• Set-PvsvmDBConnection (Machine Identity Service)

Setting the DB connection string(s)

• The database connection string is updated to the specified value regardless of

whether it is valid or not

• Specifying an invalid connection string will prevent a service from functioning

until the error is corrected

• Once a connection is configured, it is not possible to alter it without first

clearing it (set the connection to $null)

• When connection is set to $null it is reset and service is returned to an idle

state90

Viewing Host Details through PoSHHost Service PoSH provider

• Name: Citrix.Hypervisor

• Drive: <XDHyp>

○ pseudo-drive

• Two Directories

○ Connections

○ HostingUnits

PS XDHyp:\connections> dir

• Displays detailed information about each defined host connection

PS XDHyp:\hostingunits> dir

• Displays detailed information about each configured host unit

Checking IntelliCache status UselocalStorageCaching

• set-item -path xdhyp:\hostingunits\res* -UseLocalStorageCaching $true/$false

Troubleshooting Example

Error Removing Controller from SiteTroubleshooting Example 2

Check Controller status in Desktop STudio

Use PoSH to check Controller stateGet-BrokerController

Enable Desktop Studio Logging

• Edit MmcSnapin.dll.Config under C:\Program Files\Citrix\Desktop Studio○ Enable Log To File mechanism “LogFileName”

What we see…

Inner exception:

System.Net.Sockets.SocketException No such host is known

at System.Net.Dns.GetAddrInfo(String name)

at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)

at System.Net.Dns.GetHostEntry(String hostNameOrAddress)

at Citrix.Console.Models.ServiceStateModel.<GetAddressesWithExclusions>b__3(ServiceInstanceStatus address)

at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

at System.Linq.Enumerable.<SelectManyIterator>d__31`3.MoveNext()

at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()

at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)

at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

at Citrix.Console.Models.ServiceStateModel.GetAddressesWithExclusions(IEnumerable`1 addressesToExclude)

at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<GetWorkingServiceInstanceSet>d__25.MoveNext()

at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<ExecuteNext>d__c.MoveNext()

at Citrix.Console.PowerShellInteraction.PowerShellScript`1.<ExecuteNext>d__9.MoveNext()

at Citrix.Console.InteractionCore.ScriptBase.<ExecuteImplementation>d__1.MoveNext()

at Citrix.Console.InteractionCore.Scheduler.ExecuteNext(ScriptExecution execution)

What we see…

Inner exception:

What we see…

Inner exception:

Researching the Exception…www.google.com

• Exception Type

○ SocketException

• Condition

○ An error was encountered when resolving the hostNameOrAddress parameter

• Sources

○ http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx

○ http://

msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx

Researching the Exception…www.google.com

• Exception Type

○ SocketException

• Condition

○ An error was encountered when resolving the hostNameOrAddress parameter

• Sources

○ http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx

○ http://

msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx

CDF Output

Checking Service StatesGet-<Service>ServiceStatus

E.G Running Get-BrokerServiceStatus against DDC56.glover.com confirms

what the output from the CDf trace is telling us i.e. all services on the controller

we want to remove from the site have been reset:

Note: -AdminAddress parameter is used to specify which controller the cmdlet is

run against (built-in PoSH remoting capability…

Validating Service StatusReference

• Use Get-BrokerServiceStatus (Broker Service)

• Use Get-ConfigServiceStatus (Configuration Service)

• Use Get-HypServiceStatus (Host Service)

• Use Get-AcctServiceStatus (AD Identity Service)

• Use Get-ProvServiceStatus (Machine Creation Service)

• Use Get-PvsvmServiceStatus (Machine Identity Service)

Checking Registered Services

• As each service on DDC256.glover.com has been reset to DBUnconfigured,

running Get-ConfigRegisteredServiceInstance against the same controller will

return the following expected error:

Checking Service Instances

Checking Registered Services con’t

Checking Registered Services con’tGet-ConfigRegisteredServiceInstance –ServiceType

Step 1Manually cleaning up the database & removing orphan DB entries

• Take note of ServiceAccountSid used by service…

• Note: This is the objectSid of the machine the service is running on

ADExplorer

• ADExplorer can be used to

retrieve the ServiceAccountSid

assigned to the orphan services

• objectSid = ServiceAccountSid

Step 2Manually cleaning up the database & removing orphan DB entries

• Create DB eviction script for each service to fully remove them from the site…

Step 3Run config service eviction script against site database

Step 4Run the rest of the service eviction scripts against the Site DB

Step 5 Refresh Desktop Studio

• Problematic controller no

longer appears in Desktop

Studio…

Step 6The clean up…

• AD based Registration

• Run Set-ADControllerDiscovery –sync

• Registry based registration

• Update ListOfDDCs registry value via AD Group Policy

• Computer Policy\Controllers

Resources

ToolsTracing & Logging

• CDFControl

• CDFCheck 1.0 - Cmd

• CDFMarker On Demand - For XenApp and XenDesktop

• LogEnabler

ToolsMonitoring & Collection

• Citrix Scout

• XenDesktop Site Checker

• HDX Monitor 2.0

ToolsDatabase and Services

• XDPing

• XDDBDiag for XenDesktop

Before you leave…

• Conference surveys are available online at www.citrixsummit.com starting Thursday, 18 October○ Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Monday, 29 October, from your My Organiser tool located in your My Account

SUM303: XenDesktop Advanced Troubleshooting

Documents

Advanced ICD Troubleshooting: Part II - PacerICD.compacericd.com/documents/ARTICLES/Advanced ICD Troubleshooting P… · Advanced ICD Troubleshooting: Part II ... In this case, stability

ADVANCED PID TROUBLESHOOTING

Citrix TechEdge 2014 - Citrix Group Policy Troubleshooting for XenApp and XenDesktop

Advanced Troubleshooting

Vmware Advanced Troubleshooting Performance

CA Unified Infrastructure Management xendesktop …docs.nimsoft.com/prodhelp/es/Probes/Catalog/xendesktop/2.1/...6 xendesktop Guide Appendix A: Troubleshooting for xendesktop Probe

Nimsoft Monitor xendesktop Guidedocs.nimsoft.com/prodhelp/zh/Probes/Catalog/xendesktop/2.0/...6 xendesktop Guide Appendix A: Troubleshooting 41 ... For information on configuring PowerShell

Advanced rac troubleshooting

Advanced Audio Troubleshooting

SUM303: XenDesktop Advanced Troubleshooting Mick Glover Senior Support Readiness Specialist October 16, 2012

Citrix Group Policy Troubleshooting for XenApp and XenDesktop

VAX 7000 Advanced Troubleshooting

Advanced BSOD Troubleshooting

XenApp & XenDesktop Troubleshooting Assessmentcdn.goliathtechnologies.com/.../TroubleshootingAssessment-2014-04... · XenApp & XenDesktop Troubleshooting Assessment. ... Goliath retained

Advanced SAN Troubleshooting - balios.netstor.balios.net/Divers/FC/BRKSAN-3708 - Advanced SAN... · Advanced SAN Troubleshooting BRKSAN-3708. ... Troubleshooting Device connections

SUM408-Configuring and Troubleshooting XenDesktop Sites

Maintaining and Troubleshooting your XenDesktop 7.5 Environment

Ramon Scott – Lead Escalation Engineer Configuring & Troubleshooting XenDesktop Sites SUM408 May 2013

Advanced CSS Troubleshooting

Advanced troubleshooting - Lexmarkpublications.lexmark.com/publications/pdfs/e320/eng/advanced.pdf · Advanced troubleshooting Paper handling Maintenance Troubleshooting Administration