Project MedEvac 1.0

CITRIX MEDEVAC

GLOBAL ESCALATION TEAM

2 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.

Introducing Citrix Medevac 1.0

• What is Citrix Medevac 1.0?

• How does it work?

• Architecture

• How can Citrix Medevac help me?

• Case Study

• When can I use it?

• Limitations


What is Citrix Medevac 1.0?

• Citrix Medevac 1.0 is a troubleshooting Utility developed by the Citrix Escalation Team to aid in troubleshooting and resolving complex and time consuming issues that are XML, Terminal Services, IMA and RPC related

• It is a standalone Utility that does not require a Windows Service or Server Side components to be installed for its use

• It can be run from any machine on the network


What is the Health Check Agent?

• The Citrix Health Check Agent runs as a service on each Citrix Server

• This Agent has the ability to perform actions based on test results

• The tests monitor for the following: XML ticketing issues IMA Service application resolution issues Terminal Services health

• This tool is extensible by administrator to be configurable for monitoring other custom events.


ARCHITECTURE

How does it work?

A Windows 32-bit Utility used to simulate the ICA client browsing requests to test XML,Terminal Services,IMA & RPC functionality

Uses method of Isolation based on Citrix Presentation Server Farm architecture to isolate Farm functionality


“Medevac”

ARCHITECTURE

XML BrokerXML Broker

IMA Service

TermSrv

XML Service

Server Farm

Request the Least Loaded Server

DirectLL ServerQueries

RPC

Zone Data Collector

LHCDS

IMA


ARCHITECTURE…

XML TICKETING TEST:

By simulating a client connection request we are able to test XML ticketing using Medevac as if it were an actual ICA client.

RPC TEST:

Based on input provided by the user, RPC tests can be verified by making RPC calls to the specified Server testing its functionality and returning values based on the result.

TERMINAL SERVICES TEST:

By querying information from Terminal Services remotely using Medevac, we are able to test the responsiveness of Terminal Services to ensure that it is functioning properly.


IMA (Local DataCollector)

XML Service

IMA (Remote DataCollector)

Web InterfaceWeb Server

XML Broker

Or

XML Least Loaded Server Request

XML – Launching an Application.


The Least Loaded Server

Termsrv

wsxica.dll

CCTicket.dll

CCTicket.dll

XML Service

CCTicket.dll

XML Ticket

Request

IMA Service

CCTicket.dll

or

XML IMA Service

XML Broker

Web InterfaceServer

XML Ticket

Request

credentialsticket

credentialsticket

credentialsticket

XML – Launching an Application.


ARCHITECTURE

The Least Loaded Server Tests:

Performs XML, RPC & Terminal Services tests• It uses the XML Broker’s XML service to request the least loaded

farm server for a specified published application• It uses that least loaded server IP address to make health checks

directly against the least loaded server out of the servers that have that published application available

• This is a more efficient method for checking the availability of critical environment applications rather than a “shotgun” approach

Example: If your environment has a couple of hundred servers it becomes increasingly difficult to isolate a problem server

• More importantly, this utility can proactively find a problem least loaded server before users do, reducing user outages and help desk calls


ARCHITECTURE

What is actually verified on the listed server farm:

The XML service

The IMA service

The Termsrv Service

The RPC service


Enhancements for Medevac 2.0

• Save test configuration to a configuration file.

• Save and import a server lists.

• Command line parameter implementation.

• Application name browsing.

• E-Mail alerts (SMTP).

• Full ICA Connection test.

• Multi-threaded.

• Allow administrators to restart certain services. (Maybe)


How can Citrix MedEvac help me?

• Imagine a situation where you have an outage in your entire Citrix Farm during peak production time. None of your users can launch their published applications

• Everything else seems to check out fine

• Where would you begin?

• Any time wasted has severe business impact

Your Solution:

CITRIX MEDEVAC 1.0


Typical “Black Hole” Problem

Symptoms

• Server cannot accept user connections

• Server still responding to Data Collector’s heart beats

• Data Collector views server as “least loaded”

• Data Collector directs incoming user connections to this “least loaded” server

• Denial of Service occurs to users connecting to these problematic servers.


“Black Hole” Problem(s)

DataCollector

Problematic & “Least Loaded”Presentation Server

IMA TermSrv

XML

“Heart Beat”

IMA

Healthy Presentation Server

IMA TermSrvXML

Server Load Levels

“Heart Beat”

XML

User Connections


Medevac 1.5 Demo

Demo


CASE STUDY

USERS UNABLE TO CONNECT TO THEIR PUBLISHED RESOURCES


CASE STUDY

PROBLEM DEFINITION:

• Users are unable to connect to published resources in the Farm

ENVIRONMENT:

• CPS Version: Citrix Presentation Server 4.0

• CPS Hotfixes: PSE400W2K3R01

• OS Version: Windows Server 2003 Enterprise/wSP1

• Data Store: Microsoft SQL 2000/wSP3a

• CPS Servers: 50

• Zones: 1

• Connection Method: Web Interface 4.0

• ICA client: 9.x


CASE STUDY

TROUBLESHOOTING METHODOLOGY:

THE APPROACH:

Description of issue:

User attempt to launch a published application and receives an error - "Server - unavailable" or "Cannot connect to the Citrix MetaFrame Server. I/O error"

Some things we know:• Issue is affecting all users• Issue has never occurred before in this Environment• Nothing has changed in the Environment• Users are connecting through Web Interface 4.0• Web Interface does enumerate the published resources• Connecting directly to a Server works

Now Lets take a closer look at an ICA connection….


IMA (Local DataCollector)

XML Service

Problems Launching An Application (Step 1)

Problems Launching An Application (Step 1)

IMA (Remote DataCollector)


XML Broker

Or

XML Least Loaded Server Request

Local Host Cache

Dynamic Store


The Least Loaded Server

Termsrv

wsxica.dll

XML IMA Service

CCTicket.dll

credentialsticketcredentialsticket

credentialsticket

CCTicket.dll


XML Ticket Request

Problems Launching AnApplication (Step 2)

Problems Launching AnApplication (Step 2)

XML Broker

End User Client

Connection / Logon To The Citrix Server’s

Published Application


CASE STUDY

TRADITIONAL APPROACH:

• QFARM

• Zone Isolation

• Isolation based on application hosting (which servers host certain apps)

• Direct server connections

• Manual testing using custom ICA connections

• Forced ZDC elections…

• Etc.


CASE STUDY

INSTEAD…

Let’s use MedEvac to assist with troubleshooting

• Using MedEvac on a workstation from the same Domain (with Admin privileges) we can begin


CASE STUDY

• Due to the architecture of Load Balancing, we can query the XML Broker to determine the Least Loaded Server....and run some basic tests against the Least Loaded Server that is returned….

XML Broker

Least Loaded Server

Tests run against the

LLS


CASE STUDY

Now we can run further health checks against the Least Loaded Server that was returned…

• So now let us run a XML Ticketing Test, as we have already tested Terminal Services and RPC


CASE STUDY

• XML TICKETING TEST

Least Loaded Server specified

Test results

XML Ticketing

Test specified


CASE STUDY

• MedEvac identified the “Least Loaded Server” so we know which server the ZDC is routing connections to…

• So how do we verify the IMA service is functioning properly on the “Least Loaded Server?”

• By running the LLS test and specifying the Least Loaded Server in MedEvac, we are making the LLS use it’s IMA service to communicate to the Zone Data Collector to return itself as the Least Loaded Server…

• If this test returns success, we know that the IMA service on the Least Loaded Server is working properly…


CASE STUDY

LLS Test against the Least Loaded Server (IMA TEST)

FAILURE


CASE STUDY

• In this scenario, the Least Loaded Server test failed when run against itself.

• So now we can then focus all troubleshooting towards that server’s IMA service…

• Investigation indicated that the IMA service on that server was “hung” since it could not be stopped or re-started

• Although IMA still responded to the Data Collector’s “heartbeat” messages, it could not accept any connections due to its “hung” state


CASE STUDY

RESOLUTION:

• Using PSKill from http://www.sysinternals.com we manually forced termination the IMA service

• Recreated the local host cache

• Restarted IMA - started successfully

• Users able to launch their published resources as normal now


CASE STUDY

ADDITIONAL INFORMATION:

• Detailed information on Citrix Medevac and its usage can be found at the following locations:

http://support.citrix.com/article/CTX107935


When can I use it?

• The Citrix Medevac Utility can be used anytime where isolating a problem Server is difficult (with respect to load balancing, XML, Terminal Services and RPC)

• To troubleshoot connectivity issues with published resources, such as the mysterious ‘black hole’ issue

• To run direct health checks against Citrix Presentation Servers


KNOWN LIMITATIONS

• Please see the ‘Readme’ and requirements posted on the Citrix Online Support Knowledgebase

http://support.citrix.com/article/CTX107935

• Recommended to register to the Citrix Developers Network for access to Tools and Utilities

Disclaimer• These software applications are provided to you as is with no representations, warranties or conditions of

any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the code be used to support of ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the code.


Questions?


Documents

Project MedEvac 1.0