XenApp 6 Case Studies and TroubleshootingRick Berry, Escalation EngineerMark Callahan, Escalation EngineerMay 24th, 2011

• Case study for UPM issue on XenApp 6

• Case study on XenApp 6 filtered policy issue

• Questions and wrap-up

Agenda

Case study for UPM issue on XenApp 6

• Customer was experiencing hung sessions at logon

• Some users could log in, others could not

Problem Definition

• “Black Hole”

• User Profile Manager process still running

• Logged in users would eventually be affected

Citrix Confidential - Do Not Distribute

Symptoms

Citrix User Profile Manager

Functional Overview - Logon

Local Windows Devices

XenApp Servers

XenDesktop Streamed/Delivered Desktops

Profile management ServiceFile Servers

Profiles stored via File Share My Settings

Functional Details

GPO\User Configuration\Windows

Settings\Folder Redirection\My

Documents

My Documents

\profiles\UserName\

\\server\UserHome\

Folder Redirection via Network

XenApp Server[User Logon Event Location]

Active Directory

Collect

Configuratio

n

File Server

File Server

Intelligent Sync

Profile management Service

User Logon

\HKLM\Software\Policies\Citrix\UserProfileManager.

• Complete System Dump

• PerfMon

• User Profile Manager Logs

Citrix Confidential - Do Not Distribute

Troubleshooting Methodology

• Examine Kernel memory

• Examine Winlogon process

Citrix Confidential - Do Not Distribute

Troubleshooting MethodologyComplete System Memory Dump

• Performance Monitor – monitor User Profile Manager and Winlogon threads

Troubleshooting MethodologyPerformance Monitor

NORMAL

PROBLEM

Troubleshooting MethodologyUser Profile Manager Logs

[PID];WaitUntilChangeJournalIsProcessed: Waiting to finish change journal processing of partition: C

Ah Ha! A suspicious log entry!

NTFS JournalingEvent NTFS file system action

Initial write operation The NTFS file system writes a new USN record with the USN_REASON_DATA_OVERWRITE reason flag set. For more information on possible reason flags, see the USN_RECORD structure.

Setting of the file time stamp

The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE.

Second write operation The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record.

File truncation The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION.

Close operation If the user making changes is the only user of the file, the NTFS file system writes a new USN record with the following flag setting: USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION | USN_REASON_CLOSE.

• NTFS change journal was showing an increased size of the identification field.

Troubleshooting Methodology

SCREENSHOT

• Based on the data learned from the NTFS change journal examination, a code change was made to handle changes to the size of the Update Sequence Number record and a hotfix was developed.

Resolution

Resources discussed

Citrix Profile Manager Edocs Site

Citrix Profile Manager Logon Diagram

Citrix Profile Manager Logoff Diagram

CTX119791- Profile Management FAQ

CTX12559- Citrix Profile Manager Upgrade FAQ

CTX124455- How to Capture CDF Startup Traces on UPM 3.0

Resources – Citrix Profile Manager

Log Parser for Citrix Profile Management

Memory Dump File Not Being Generated on Provisioned Target

Microsoft Windows Change Journals

Resources – Citrix Profile Manager

Case study on XenApp 6 filtered policy issue

• Customer had a new XenApp 6 farm in place• XenApp 6 Citrix policies (both computer and user settings) were being applied via Active Directory Group Policy

Objects (GPOs)• Some of the Citrix policy settings were filtered for Access Gateway connections and others were filtered by client IP• When end users connect to the XenApp 6 server from an Access Gateway site, the filtered policy settings were not

applying to the session

Problem definition

XenApp 6 policies overview

• Manage XenApp servers collectively by grouping servers into worker groups

• You can assign published applications and Citrix policies to worker groups

• Servers added to worker groups inherit settings

XenApp 6 Group-based administration

Worker Group 1

Worker Group 2

XenApp Farm

Published Application:Notepad.exe

Citrix Policy:Enable Client Drive Mapping

• Worker Group is a new filter for applying Citrix policies

• Automatic configuration of new XenApp servers by placing them in an existing worker group

Applying Citrix Policies to Worker groups

Citrix policy creation and administration

1. Create policies as Citrix IMA-based policies using Delivery Services Console (Used if AD does not exist or access is limited)

2. Create policies as Active Directory-based policies using Group Policy Management Console (GPMC)

Note: All Citrix policy settings are configurable using either administration method

• Citrix policies added via the DSC are stored in the datastore

• Two types of policies categorized by computer policies and user policies

• Can be “filtered” for granular control or “unfiltered” to apply to all servers or users

• Policy settings are stored in the servers registry

Citrix policies via the Delivery Services Console

• Filtered policy• Applies to specific group of users or servers• Uses a variety of filters (IP, AG, Groups, Client name)• Use case: Disable CDM for the Marketing domain group

• Unfiltered policy• Applies to all servers or users• Used when filters or granular control isn’t necessary• Use case: Specifying the license server that all farm servers will use

Filtered versus unfiltered policies

• Allows integration of Citrix policies into the Windows GPO engine

• Adds a Citrix node in the Group Policy Management Console and Group Policy Object Editor

• Installed with Delivery Services Console• Must be installed on the same machine where

Group Policy Objects are administered• Can be installed on a standalone machine used for

administrative purposes

Citrix policy extension

• Computer policies • Enables or disables server settings that were once under the farm and server

properties in previous versions• Registry location:

• 32-bit components: HKLM\Software\Policies\Citrix

• 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix

• User policies • Enables or disables specific features for user sessions• Registry location:

• 32-bit components: HKLM\Software\Policies\Citrix\<SessionID>

• 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix\<SessionID>

Citrix policy settings on the server

GPO processing and precedence

Local PoliciesLocal Policies

Citrix Group Policy ObjectsCitrix Group Policy Objects

Site Group Policy ObjectsSite Group Policy Objects

Domain Group Policy ObjectsDomain Group Policy Objects

OU Group Policy ObjectsOU Group Policy Objects

PR

OC

ES

SIN

G

PR

EC

ED

EN

CE

Citrix policies general roubleshooting checklist

Identify how the policies are being applied (e.g. Active Directory, DSC, both)?

Are the Citrix policy files present on the server?

What does the group policy results wizard show?

CDF Tracing results (see CTX113199 for modules).

Setup and review Citrix policy debugging logs.

Are the Citrix policy registry settings in place?

Troubleshooting Methodology

• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

Troubleshooting methodology for the case

• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

• What does output from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.

• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?

• What does out from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.

• Enable Citrix policy debugging (see CTX128413)

Troubleshooting Methodology

Setting these values to 0xFFFFFFFF writes the debug information to a log file: %SYSTEMROOT%\Temp\CitrixCseEngine.log

Setting these values to 0x0000FFFF writes the debug information to a debugger such as DebugView

NOTE: The same values have to be written to HKLM\SOFTWARE\Wow6432Node\Citrix\GroupPolicy

For more details see CTX128413

• Reviewing %SYSTEMROOT%\Temp\CitrixCseEngine.log we need to verify the logged in user

User Name = REDGETLAB\rickbeuser1, SID = S-1-5-21-3992822370-2973014269-1922904879-1172, Session ID = 3

Computer Identity - Name = 60426497M1

• Next we search on the display name of our policy so we can get the GUID since the GUID is referenced more in the log

Name={52243C73-ED52-4539-B484-02098F5A88F4}, DisplayName=Test Policies, Link=LDAP://OU=RickBe,DC=REDGETLAB,DC=CTX

Troubleshooting Methodology – Debug logs

• We know that the Access Gateway filter on the policy was using a wildcard (apply to any Access Gateway site), so for the Access Gateway filter we can search on AGInUse

FullArmor.GroupPolicyFramework:And(Citrix.Policy.Templates:AGInUse.isValid, Citrix.Policy.Templates:AGFarm.isValid

Citrix.Policy.Templates:WildcardMatch("*"

Citrix.Policy.Templates:AGTags.value,"*",true

Troubleshooting Methodology – Debug logs

• Our session in question was session 3:

HKLM\SOFTWARE\Policies\Citrix\3\Events

"LastUpdate"="2011-03-27 04:12:12Z“

• Looking at the Evidence key:

HKLM\SOFTWARE\Policies\Citrix\3\Evidence

“AGFarm”=

"AGInUse"=dword:00000000

Troubleshooting Methodology – Registry review

These are issues!!

• Reviewing the debug logs and comparing this to the registry entries being made allowed us to narrow down the issue to how the policy filters were being evaluated

• Through our analysis it was determined that there was an issue with the filter expression logic when the Access Gateway filter was being used

Root cause isolation

• The investigation into this issue resulted in code change for the Delivery Services Console which was tested successfully by the customer

• This code change is currently being packaged into a hotfix for the Delivery Services Console

Resolution

Resources discussed

CTX125152 - Citrix Group Policy Engine Facts in XenApp 6

CTX127612 - How Policies are Applied when an ICA Session Connects to XenApp 6.0

CTX127611 - How Citrix IMA Policies for XenApp 6.0 Fit in to Microsofts GPO Processing and Precedence Model

CTX124241 - Technical Guide for Upgrading/Migrating to XenApp 6

Citrix Blog Site - XenApp 6 Policies Deep Dive

Resources – Citrix Policy Architecture

CTX128413 - XenApp 6 and XenDesktop 5 Group Policy Tracing

CTX111961 - CDFControl Tool

CTX113199 - IMA Modules to Select When Obtaining a CDF Trace for a Policy Problem

Resources – Citrix Policy troubleshooting

Questions?

Before you leave…

• Session surveys are available online at www.citrixsummit.com starting Thursday, May 26• Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account