Server Virtualization in Microsoft Lync Server 2010

Server Virtualization in Microsoft Lync Server 2010

Last updated 2/18/2011

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. Copyright© 2011 Microsoft. All rights reserved.Microsoft, Windows, Windows Media, Windows PowerShell, Windows Server, Active Directory, Forefront, Hyper-V, Lync, and SQL Server are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.

Table of ContentsSummary.....................................................................................................................................................1

Glossary.......................................................................................................................................................2

1 Introduction.........................................................................................................................................5

1.1 What is Virtualization?.................................................................................................................5

1.2 Why Virtualize?............................................................................................................................5

2 Virtualization in Lync Server 2010.......................................................................................................6

2.1 Terminology.................................................................................................................................6

2.2 Supported Functionality..............................................................................................................7

2.3 Hypervisors..................................................................................................................................8

2.3.1 Validated Hypervisors..........................................................................................................8

2.3.2 Other Hypervisors................................................................................................................8

2.3.3 Unsupported Hypervisors....................................................................................................8

2.4 Guest Operating System Requirement........................................................................................8

2.5 Virtualized Lync Server Deployment General Considerations......................................................8

2.5.1 Mixing Physical and Virtual Servers Running Lync Server 2010...........................................8

2.5.2 Spreading Lync Server Workloads on Different VM Hosts...................................................9

2.5.3 High Availability through Virtualization...............................................................................9

2.5.4 Quick Migration and Live Migration Support.......................................................................9

2.5.5 Dynamic Memory Support...................................................................................................9

3 Typical VM Host Configurations.........................................................................................................10

3.1 VM Host Server Configuration...................................................................................................10

3.2 VM Guest Operating System and Application Storage Configuration........................................11

3.3 Small VM Host Configuration.....................................................................................................11

3.4 Medium VM Host Configuration................................................................................................12

3.5 Large VM Host Configuration.....................................................................................................13

3.6 VM Host Networking Considerations.........................................................................................15

4 Virtualization Configuration Guidance...............................................................................................15

4.1 Applications Running in Host Root Partition..............................................................................15

4.2 Guest Virtualized Storage..........................................................................................................16

4.2.1 Disk Types..........................................................................................................................16

4.2.2 Mapping of Storage to VM Guests.....................................................................................16

4.2.3 VM Guest Storage Recommendations by Workload..........................................................16

4.3 Other VM Guest Configuration Settings.....................................................................................18

4.3.1 Guest Operating System....................................................................................................18

4.3.2 Network adapter................................................................................................................18

4.3.3 Ratio of Physical to Virtual CPUs........................................................................................18

4.3.4 Support of VM Image Templates.......................................................................................19

4.3.5 Disable Virtual DVD/CD Drives in Guest Operating System................................................19

5 Scale Comparison of Virtualized and Physical Lync Server Roles.......................................................19

5.1 CPU............................................................................................................................................19

5.2 Memory.....................................................................................................................................19

5.3 Scale Comparison Table Physical-Virtual...................................................................................20

5.3.1 Maximum Conference Size Physical-Virtual.......................................................................21

5.3.2 Stand-alone A/V Conferencing Server Recommendation Physical-Virtual.........................21

6 Sample Lync Server Virtualized Topologies.......................................................................................21

6.1 Virtual Enterprise Edition Deployment A...................................................................................21

6.2 Virtual Enterprise Edition Deployment B...................................................................................22

6.3 Mix Virtual/Physical Enterprise Edition Deployment.................................................................24

6.4 Virtual Standard Edition Deployment.......................................................................................25

7 Test Results........................................................................................................................................26

8 Measuring Performance on Virtual Lync Server 2010.......................................................................27

8.1 Performance Indicators.............................................................................................................27

8.1.1 Processor and Percentage of Processor Time....................................................................27

8.1.2 Network Interface..............................................................................................................27

8.1.3 Memory.............................................................................................................................27

8.1.4 Databases..........................................................................................................................27

8.1.5 SIP......................................................................................................................................27

Appendix A: How to Enable VMQ on Intel Network Adapter....................................................................29

Appendix B: How to Increase Send/Receive Buffers on Intel Network Adapter........................................30

Appendix C: Changing the Bandwidth Limit for Application Sharing.........................................................31

Appendix D: Performance Counters..........................................................................................................32

1 Lync Server 2010 Performance Counter Reference...........................................................................32

1.1 All Servers..................................................................................................................................32

1.2 Front End Servers.......................................................................................................................33

1.3 Edge Server................................................................................................................................36

1.4 Back End Server.........................................................................................................................38

1.5 Mediation Server.......................................................................................................................39

1.6 Audio/Video Conferencing Server.............................................................................................40

1.7 Establishing Performance Baselines...........................................................................................40

1.7.1 Resource Utilization Baseline Tests....................................................................................40

1.7.2 User Load Baseline Tests....................................................................................................41

1.7.3 Multipoint Conferencing Unit (MCU) Resource Utilization Baseline Tests: (Monitor on All MCUs) 41

SummaryVirtualization is a powerful technology that enables IT departments to reduce costs and improve operational efficiency. Virtualization is increasingly being seen as a way to allocate resources smartly, and consolidate environments. Building on the efforts taken in Microsoft® Office Communications Server 2007 R2, Microsoft® Lync™ Server 2010 communications software provides a wide range of server virtualization capabilities to reduce total cost of ownership (TCO). This document outlines a series of best practice recommendations for running Lync Server 2010 in a virtualized environment.

1


FiguresFigure 1 Small VM host configuration diagram...................................................................................................2Figure 2 Medium VM host configuration diagram..............................................................................................2Figure 3 Large VM host configuration diagram...................................................................................................2Figure 4 Virtual Enterprise Edition deployment A...............................................................................................2Figure 5 Virtual Enterprise Edition deployment B...............................................................................................2Figure 6 Mixed virtual-physical Enterprise Edition deployment..........................................................................2Figure 7 Virtual Standard Edition deployment....................................................................................................2Figure 3 Advanced tab of Network Adaptor Properties, showing Virtual Machine Queue setting........................2Figure 4 Advanced tab of Network Adapter properties, showing Receive Buffers property.................................2

Tables Table 1 Supported virtualized Lync Server roles and workloads..........................................................................2Table 2 VM host configuration categories..........................................................................................................2Table 3 VM host hardware base configuration...................................................................................................2Table 4 VM guest storage configuration............................................................................................................. 2Table 5 VM guest virtual storage options........................................................................................................... 2Table 6 VM guest storage sizing recommendations............................................................................................2Table 7 Virtual-physical scale comparison table.........................................................................................................2

GlossaryThe following are phrases and acronyms used throughout this document.Phrase/Acronym MeaningStandard Edition server Lync Server 2010, Standard Edition is designed for small organizations,

for pilot projects, or for branch sites within larger organizations. It enables all of the features of Lync Server 2010, in addition to the necessary databases, to run on a single server. Lync Server 2010 Standard Edition does not provide a true high availability solution.

Front End Server Lync Server 2010, Front End Server is the core server role of an Enterprise Edition pool, and performs the basic Lync Server 2010 functions. It is one of two server roles required to be in any Lync Server 2010, Enterprise Edition deployment.

Enterprise Edition pool Lync Server 2010 Enterprise Edition is a collection of Front End Servers. Enterprise pools with two or more Front End Servers provide high availability, and allow scale out to a total of 10 Front End Servers within a pool. Enterprise pools allow separation of A/V Conferencing Server components.

Back End Server Lync Server 2010, Back End Server is a core server role, and hosts the Microsoft® SQL Server™ databases used by the Lync Server Front End Server. The Back End Server is the second of two server roles required to be in any Lync Server 2010 Enterprise Edition deployment. Back End Servers can be deployed on a single server running SQL Server, or on a SQL Server cluster.

2


Pool A pool is a set of identically configured servers running Lync Server that work together to provide services for a common group of users. A pool provides scalability and resiliency for the hosted workloads and functions.

A/V Conferencing Server Lync Server 2010, A/V Conferencing Server provides audio/video (A/V) conferencing functionality to a deployment. It can be collocated with the Lync Server 2010 Front End Server, or deployed separately as a single server or Lync Server 2010 A/V Conferencing Server pool. For best performance, A/V Conferencing Server should be deployed separately to the Front End Server. Standard Edition server cannot utilize a stand-alone A/V Conferencing Server. AV Conferencing Servers are deployed as a single server or a pool of servers. No more than one A/V Conferencing pool is supported per site.

Edge Server Lync Server 2010, Edge Server enables users to communicate and collaborate with users outside an organization’s firewalls.Edge Servers are deployed as a single server or a pool of servers.

Monitoring Server Lync Server 2010, Monitoring Server collects data about the quality of network media in both Enterprise Voice calls and A/V conferences, as well as call detail records (CDRs). Monitoring Server is typically collocated with the Microsoft Lync Server 2010, Archiving Server. A SQL Server Back End Server is required to implement a Monitoring Server. Monitoring Server can be deployed only as a single server.

Archiving Server Lync Server 2010, Archiving Server enables the archiving of instant messaging (IM) communications and meeting content for compliance reasons.A SQL Server Back End Server is required to implement Archiving. Archiving Server can be deployed only as a single server.

Director The Microsoft Lync Server 2010, Director can authenticate Lync Server requests and redirects users to the their respective home pool. The Director role is most useful in deployments that enable external user access where the Director can authenticate requests before sending them on to an internal server. Directors can also improve performance in organizations with multiple Front End pools. Directors are deployed as a single server or a pool of servers.

Application Sharing service The Application Sharing service, hosted on a Front End Server.

Mediation Server The Microsoft Lync Server 2010, Mediation Server is a necessary component for implementing Enterprise Voice and dial-in conferencing. Mediation Server translates signaling and, in some configurations, media between your internal Lync Server infrastructure and an public switched telephone network (PSTN) gateway or a SIP trunk. Mediation Server can be collocated with the Front End Server, or can be installed as a stand-alone server for better performance. A stand-alone Mediation Server is recommended if Media Bypass is not available. Stand-alone Mediation Servers are deployed as a single server or a pool of servers.

3


Multi-modal Conference A conference using all modalities of Lync Server, including: application sharing, Voice over Internet Protocol (VoIP), video, instant messaging, dial-in conferencing, and web conferencing.

VM A virtual machine is an instance of a guest operating system (running Windows Server® 2008 R2) running on a server running a hypervisor, such as Microsoft® Hyper-V™ or VMware ESX.

Site Lync Server 2010 sites are defined as a location-based or logical grouping of collection of Lync Server 2010 components. A site is a set of computers that are well connected by a high-speed, low-latency network, such as a single local area network (LAN) or two networks connected by a high-speed fiber optic network. Note that Lync Server sites are a separate concept from Active Directory® Domain Services (AD DS) sites and Microsoft® Exchange Server sites. Your Lync Server sites do not have to correspond to your Active Directory sites.

Hypervisor A native hypervisor runs directly on the host hardware on the lowest level (Ring -1, below kernel mode), and provides abstraction to allow multiple operating systems to run concurrently on a host computer, a feature called hardware virtualization. The hypervisor provides the guest operating systems with a virtual operating platform and monitors the execution of the guest operating systems.

4

http://en.wikipedia.org/wiki/Platform_virtualization


1 Introduction

1.1 What is Virtualization?

Virtualization, in its simplest form, provides a layer of abstraction between the hardware and operating system, and allows multiple operating system instances to share the same hardware. Server virtualization allows multiple operating systems (or “virtual machines”) and applicable workloads to share the resources of a single server. The main benefits of virtualization are improved resource utilization, and the ability to relocate virtual machines to different hosts, typically reducing the number of physical servers needed.

1.2 Why Virtualize?

In today’s competitive environment, organizations are increasingly dealing with concerns around reducing infrastructure costs while accurately forecasting operational expenditure, creating more value by providing faster but lower-cost services, and increasing operational efficiency at the same time. A well-managed IT environment that is efficient, reliable, and easily managed is essential to successfully meeting the above goals of the organization. Lync Server, with its server virtualization capabilities, helps companies achieve this goal by providing the following benefits: Reduced Total Cost of Ownership

o Leverage existing resources with server consolidationo Increase space utilization with reduced number of physical serverso Reduce power consumption and cooling requirementso Decrease hardware maintenance and training costso Provide an abstraction of operating system and workloads from actual physical hardwareo Reduced Time to deploy o Minimize service disruption and increase service levelso Provide capacity on demand in response to changing business needso Speed up server deployment by duplicating a standard virtual server

Increased operational efficiencyo Improve business continuity with high availability of hardware and applicationso Increase operational flexibility with dynamic resource management and server provisioning o Improve server to admin ratio

5


2 Virtualization in Lync Server 2010Microsoft has made significant investments in providing extensive virtualization support for Lync Server across a fully distributed topology. Lync Server can be virtualized by using Windows Server 2008 R2 Hyper-V, or other virtualization solutions that have been tested by Microsoft. For more details, see “Windows Server Virtualization Validation Program” at http://go.microsoft.com/fwlink/?linkid=200511.

Due to limitations such as packet loss, delay, jitter, and clock skew, Office Communications Server 2007 R2 could only virtualize a few workloads such as Presence and IM. Lync Server 2010 now also supports server-side virtualization of real time media, thus enabling conferencing and voice workloads to be virtualized.

Lync Server enables the following workloads to be virtualized:

Instant Messaging (IM)

IM conferencing

Presence

Enterprise Voice (PSTN)

Audio Conferencing

Video Conferencing

Web Conferencing

Application Sharing

Remote Access, Federation (Edge Server)

Response Group service

Virtualized Lync Server deployment can be considered for all sizes of deployment: From small single-site deployments, to large multi-site enterprise implementations.

This document provides details on:

Best practice guidelines for deploying a virtualized Lync Server topology

Typical VM host configurations

VM guest configuration best practices based on the workload

Scale comparison of virtualized and physical Lync Server roles

Measuring Performance

Note: This document does not cover client virtualization scenarios such as application virtualization or remote desktop scenarios for the Lync client.

2.1 Terminology

Throughout this document, the following statements will be used:

Recommended: A recommended configuration or setting that has been explicitly validated, or follows the industry best practice standards.

6


Not recommended: A configuration or setting that is not recommended in a production deployment, based on the industry best practice standards, or that has been specifically identified as a potential problem in a virtualized Lync deployment.

Not supported: A configuration, feature, or setting that must not be implemented in a virtualized Lync Server deployment. Running unsupported deployment may negatively impact functionality or performance of a Lync Server deployment.

2.2 Supported Functionality

Table 1 lists the server roles and corresponding features that are tested and supported in a virtualized site. Role Supported Supported Features

Enterprise Front End Yes Registration, Presence, IM, Enterprise Voice, audio/video

Standard Edition Server Yes Registration, Presence, IM, Enterprise Voice, audio/video, SQL Store

SQL Server Back End Server

Yes SQL Store

File Store Yes File serverEdge Server Yes Remote Access

A/V Conferencing Server Yes Audio/videoMediation Server Yes Enterprise Voice Director Yes Authentication, RedirectionMonitoring Server Yes Monitoring service, QoE, and call detail

recordsArchiving Server Yes Archiving serviceSurvivable Branch Office Appliance (OEM appliance)

No Registration, Enterprise Voice

Survivable Branch Office Server(non-OEM server)

Yes Registration, Enterprise Voice

Table 1 Supported virtualized Lync Server roles and workloads

Like a physical Lync Server, virtual Lync Server deployments will integrate with Microsoft Exchange Server 2010 to provide a Unified Messaging solution. Deployment of Lync Server in a virtual environment is described in this white paper. For details about deploying Exchange 2010 Unified Messaging, see “Understanding Unified Messaging Deployments” at http://go.microsoft.com/fwlink/?LinkId=210839.

7


2.3 Hypervisors

2.3.1 Validated Hypervisors Windows Server 2008 R2 Hyper-V (For details about hotfix, see Microsoft Knowledge Base

article 981836, “Network connection is lost on a Windows Server 2003-based Hyper-V VM” at http://go.microsoft.com/fwlink/?LinkId=201212)

Microsoft® Hyper-V™ Server 2008 R2 Hyper-V (For details about hotfix, see Microsoft Knowledge Base article 981836, “Network connection is lost on a Windows Server 2003-based Hyper-V VM” at http://go.microsoft.com/fwlink/?LinkId=201212 )

VMware ESX 4.0

2.3.2 Other HypervisorsOther Hypervisor implementations that have been validated through the Windows Server Virtualization Validation Program.Note: Lync Server is a real-time application workload which requires high-performance I/O (Network and Storage) which some hypervisor implementations may not be able to deliver. We recommend validating a deployment by using synthetic load tools.

2.3.3 Unsupported HypervisorsAlthough Hyper-V was available as a role in Windows Server 2008, that release is not supported. A number of enhancements that are critical for running virtualized Lync Server media workloads were implemented with Windows Server 2008 R2 to address network packet loss.

2.4 Guest Operating System Requirement

Windows Server 2008 R2 is required guest operating system. Critical network performance improvements are only available in the Windows Server 2008 R2 operating system.

Windows Server 2008 SP2 is not supported as a guest operating system.

2.5 Virtualized Lync Server Deployment General Considerations

2.5.1 Mixing Physical and Virtual Servers Running Lync Server 2010

Generally physical and virtual servers can be combined in a deployment, with the following limitations:

Mixed virtual and physical servers within the same pool are not supported. A pool1 of Lync Server roles must be deployed as a homogenous set of servers (either physical or virtual) which roughly provide the same amount of resources. For virtualized deployments, ensure that the virtual machines are hosted in a way that equal resources are available to each guest in the pool. Lync Server does not perform load-based load balancing with the exception of the A/V conferencing workload. Note that a virtual Front End Server and a physical SQL Server Back End Servers are explicitly supported.

Balanced end-to end-performance is required. When sizing a virtualized Lync Server deployment, use the guidance provided in this document to size each role adequately, and perform tests using the synthetic load tools available to validate that no bottlenecks exist.

1 A pool is defined as a Lync Server pool with more than two servers including the follow Server roles: Enterprise Edition Front End Server, Director, Mediation Server, A/V Conferencing Server, and Edge Server.

8

http://go.microsoft.com/fwlink/?LinkId=201212



2.5.2 Spreading Lync Server Workloads on Different VM Hosts

If you are implemented a highly available, virtualized, Lync Server deployment consider spreading similar workloads to different VM hosts. For example, do not run all of the Front End Servers on the same VM host because you could potentially lose the complete pool if there is a failure on an individual VM host. Consider recovery of virtual machines on a different host as tool to limit the outage, but not as a tool to implement High Availability (also see next bullet).

2.5.3 High Availability through Virtualization

Virtualizing a Lync Server deployment is not a replacement for implementing High Availability: Implement a pool of Lync server components (such as an Front End pool or Edge pool with two or more servers), as well as a clustered SQL Server Back End Server in order to achieve the best possible High Availability even in a virtualized setup. Migration to the next available virtual machine host is a tool to mitigate the duration of downtime, however does not provide the best High Availability solution for Lync Server.

2.5.4 Quick Migration and Live Migration Support

Quick Migration and Live Migration with Lync Server 2010 workloads have not been validated by the product group at this point. Future updates of this white paper will provide more guidance.

Considerations for migrating Lync Server workloads:

Front End Servers: Using draining prior to migration to minimize any client impact. Clients will disconnect from active sessions, such as active conferences, during migration.

Migrating Front End Server or A/V Conferencing Server while a conference is progress: The delay will cause all clients to disconnect and the conference is recreated on another Front End Server, or A/V Conferencing Server.

SQL Server Back End Server workload: While migration of the SQL Server back end VM, the Front End Servers will switch to resiliency mode until the migration is completed, and should reconnect.

Depending on the time a migration takes, and what workloads are concurrently migrated, clients may connect to the backup registrar pool.

Live Migration of Lync Server workloads have not been validated, and is not supported (Applicable to Hyper-V Live Migration, and VMware Vmotion). Specifically, it has not been validated what the client experience would be if a live migration is performed on a Lync Server workload running an active conference.

2.5.5 Dynamic Memory Support

Windows Server 2008 R2 Hyper-V SP1 (Service Pack 1) will provide support for Dynamic Memory (for details, see “Implementing and Configuring Dynamic Memory” at http://go.microsoft.com/fwlink/?LinkId=211071). Dynamic memory has not been validated with Lync Server 2010 workloads, and specific guidance cannot be provided.

3 Typical VM Host ConfigurationsGlossary:

9




Direct Attach Storage (DAS): Disk storage local to host, not shared.

Storage Area Network (SAN): Storage devices accessible through a dedicated network.

Internet Small Computer System Interface (iSCSI): IP-based storage networking standard, typically using Ethernet.

Fiber channel (FC) : Network technology primarily for storage networking.

MPIO = Multi-path IO: Redundant network path to a shared storage device.

Virtual Machine Queue (VMQ): Hardware virtualization technology for the efficient transfer of network traffic to a virtual machine.

Gigabit Ethernet (GbE): Type (Ethernet) and speed (units of Gigabit per second) of a network infrastructure.

Serial Advanced Technology Attachment (SATA): Computer bus interface for mass storage attachment.

Serial Attached SCSI (SAS): Computer bus interface for mass storage attachment.

Solid state drive (SSD): A data storage device using solid-state memory.

Logical unit number or LUN: A number used to identify a logical unit in a SAN or other storage devices. Mostly used to refer to a logical disk in a SAN.

Hardware vendors typically provide guidance on what type of servers, storage, and networking components are recommended for production-quality VM host configurations. The following section provides a generic overview of what configurations are suitable for deploying a virtual Lync Server deployment, and what type of configurations should be avoided.

With the context of this document VM host configurations are defined in the following scale, and are outlined in more detail below

Size VM Host Configuration

Small Single VM host with direct attached storage (DAS)

Medium 4-12 VM hosts with shared storage (iSCSI SAN) multi path I/O (MPIO)

Large 12-48 VM hosts with fiber connected shared storage (FC SAN multi path I/O)

Table 2 VM host configuration categories

3.1 VM Host Server Configuration

The follow table represents a recommended hardware base configuration for a VM host as of Q4/2010.

Component Spec

Server Enterprise-grade server, with a minimum of 2 CPU sockets (or more)

CPU Intel Xeon 5500 series or AMD Opteron 6100 series (2 GHz+)

CPUs supporting NPT/EPT (Nested Page Tables/Extended Page Tables)

10

http://en.wikipedia.org/wiki/Data_storage_device


Network adapter Dual (or more) 1 GbE or 10 GbE adapter (with VMQ support is recommended for 10 GbE network adapternetwork adapter). Note: Network adapter chipsets from Intel, Broadcom, and other vendors support VMQ.

Storage Two or more SATA or SAS HDD 10k rpm or higher (DAS), or equivalent storage, RAID 1 (VM host operating system usage), or equivalent SSD

Memory 32GB+ PC2-6400 DDR2, or PC3-8500 DDR3 Memory (or faster), must be ECC compliant

Table 3 VM host hardware base configuration

3.2 VM Guest Operating System and Application Storage Configuration

Storage recommendations are based on typical enterprise-grade traditional hard disk drives, or equivalent enterprise-grade solid state drives (SSDs). Typical protocols for Storage Area Networks are iSCSI, Fiber Channel, or other equivalent solutions.

Type Spec

DAS Direct Attached Storage with SAS (15k or 10k rpm) or SATA (7.2k rpm) drives

iSCSI SAN Storage Array Network with 1 GbE or 10 GbE iSCSI storage with enterprise class SAS (15k rpm) drives, or equivalent SSD drives. 2+ GB cache.

FC SAN Fiber connected Storage Array Network with 4, or 8Gbps Fibre Channel with enterprise class SAS (15k rpm) drives, or equivalent SSD drives. 4+ GB cache.

CSV Cluster Shared Volume (CSV) is a volume that is simultaneously available to directly read from and write.

Table 4 VM guest storage configuration

3.3 Small VM Host Configuration

A single VM host with direct attached storage configuration.

11


Figure 1 Small VM host configuration diagram

Recommended networking:

Dual network adapter/host: 1 network adapter reserved for host access, 1+ network adapter for shared VM guest application workload.

network adapter speed: 1 GbE or 10 GbE (VMQ)

Recommended storage:

Dedicated spindle for VM host, as well as a dedicated spindle for each VM guest virtual drive

Not recommended:

100 Mbps network adapters

Single network adapter (for VM host and VM guest application workload)

Shared spindles between host and guests, or between guests

3.4 Medium VM Host Configuration

Multiple VM host with shared storage (SAN) attached through iSCSI.

12


Figure 2 Medium VM host configuration diagram


3 network adapters/host: 1 network adapter reserved for host access, one 1 for shared VM guest application workload, 1 (or more) dedicated to iSCSI SAN access

MPIO is recommended for network access to SAN

Separation of application and iSCSI traffic (dedicated iSCSI segment)

Network adapter speed: 1 GbE or 10 GbE.

Implement VLANs to isolate network traffic


Dedicated SAN LUN for each VM guest virtual drive (VHD or passthrough)

Cluster Shared Volume (CSV)

Not recommended:

Host with less than 3 network adapters

Shared Host/Application/iSCSI network adapter configurations

Shared LUNs for VM guest or application workloads

3.5 Large VM Host Configuration

Multiple VM host with shared storage (SAN) attached through Fiber Channel. Multiple network adapter (network adapter teaming) for load balanced guest network access. Dedicated network adapter for host-to-host communication (VM quick migration).

13


Figure 3 Large VM host configuration diagram


4+ network adaptera/host: 1 network adapter reserved for host access, one 2 network adapters for shared VM guest application workload, 1 network adapter for VM host to host communication

Fiber Channel (4 Gbps or 8 Gbps),

Network adapter speed: 1 GbE or 10 GbE (10 GbE recommended)

Guest network adapter teaming (IEEE 802.3ad, network adapter teaming). Network adapter fault tolerance, and link aggregation

Redundant network adapter and Fiber channel controllers, MPIO is recommended

Alternatively to dedicated Fiber Channel

Implement VLANs to isolate network traffic


Dedicated SAN logical unit number (LUN) for each VM guest virtual drive (VHD or passthrough)

Cluster Shared Volume (CSV)Redundant controllers in SAN

Not recommended:

Shared host/application network adapter configurations

Shared LUNs for VM guest or application workloads

14


3.6 VM Host Networking Considerations

Important Note: Lync Server media workloads, such as A/V and application sharing require more bandwidth that other typical server workloads (such as email or SQL Server database). The Lync media workloads are also more vulnerable to high latency and packet loss problems.

While designing a virtualized Lync deployment it important to take networking contention into account.

Networking considerations:

VM host must have at least one dedicated network adapter for the Lync Server workload. Shared usage with the VM host, or specifically with SAN access is not recommended.

A Lync Server Media workload (primarily Front End Servers and A/V Conferencing Servers), can reach a peak network utilization of more than 500 Mbps (0.5 Gbps).

If a specific VM host is targeted to host multiple guest VMs running Lync media workloads, ensure that the host network adapter is sized accordingly. To mitigate bottlenecks consider higher speed network adapters (such as 10 GbE), or multiple network adapters using link aggregation (network adapter IEEE 802.3ad), for example a 3 x 1 GbE network adapter in a network adapter teamed setup. For details, see Mircosoft Knowledge Base article 968703, “Microsoft Support Policy for NIC Teaming with Hyper-V” at http://go.microsoft.com/fwlink/?LinkId=211073.

Enable VLAN tagging on the host network adapter, and implement (multiple) VLANs on the guests to optimize the network traffic for the various workloads, such as Front End Server to Client/AVMCU, and Front End Server to SQL Server Back End Server.

Implement MPIO (Multi-Path IO) to your storage Back End Server.

Network adapter supporting VMQ: Enable Windows Server 2008 R2 Hyper-V and enable VMQ on the host network adapter. Note that not all network chipsets support VMQ. Current network adapter chipset vendors with VMQ support are Intel, Broadcom, and other vendors.

4 Virtualization Configuration GuidanceThe VM guest configurations described in this section are based on virtualization best practices for general enterprise scale application workloads. While concepts and implementation details discussed here are specific to Hyper-V on Windows Server 2008 R2, similar concepts and features are available with other virtualization platforms.

4.1 Applications Running in Host Root Partition

Running application in the VM host root operating system is not recommended.

If antivirus software is deployed, ensure that the necessary scanning exceptions are implemented (such as files, folders, and processes ). For details, see Microsoft Knowledge Base article 961804, “Virtual machines are missing in the Hyper-V Manager Console or when you create or start a virtual machine, you receive one of the following error codes: “0x800704C8”, “0x80070037” or “0x800703E3” at http://go.microsoft.com/fwlink/?LinkId=210844.

For further recommendations, see “Planning for Hyper-V Security” at http://go.microsoft.com/fwlink/?LinkId=210845.

15






4.2 Guest Virtualized Storage

The following table describes the different types of virtual hard drives in the guest. For best performance results, the a virtual disk has to reside dedicated physical disks, or on a dedicated logical unit on a SAN.

4.2.1 Disk Types

Type Recommended Comment

Dynamic Disk No Dynamic Disk VHDs are not recommended for production servers.

Fixed Disk Yes Fixed Disk VHDs must be placed on a dedicated spindle in Direct Attached Storage, or on a dedicated LUN (Logical Unit) in a SAN.

Passthrough Disk

Yes Passthrough disks are physical disks, or LUNs on a SAN which are exposed to the host and accessed by the guest directly. The performance is typically slightly higher than fixed disks.

Table 5 VM guest virtual storage options

For details, see “Planning for Disks and Storage” at http://go.microsoft.com/fwlink/?LinkId=210846.

4.2.2 Mapping of Storage to VM Guests

Hyper-V offers a number of different ways to expose a drive to a VM guest operating system. The appropriate method depends on a number of factors, such as if the storage is locally attached, or on a SAN, and there are no specific preferences from a Lync Server visualization perspective.

Common best practices:

VM guest operating system boot drive must be mapped as a virtual IDE drive. SQL Server workloads use a virtual IDE drive for the operating system, and SQL Server binaries, and virtual SCSI controllers to map SQL Server log and SQL Server data drives

iSCSI targets directly exposed to the VM guest is typically implemented if VMs need to be moved between VM hosts. Here the guest operating system drive on the SAN and the guest specific data drives will automatically be accessible to the guest after the move. If VMs are moved, it is required that guest storage is located on a shared storage attached using either iSCSI, for Fibre-Channel.

4.2.3 VM Guest Storage Recommendations by Workload

4.2.3.1 Guest Storage Sizing Recommendations

Storage type Size

Guest VM operating system with Lync Server installed (including paging file)

50 GB

SQL Server storage drives 30-100 GB*

16



File storage drives 30 GB – 500 GB*

Table 6 VM guest storage sizing recommendations

* Size depends on scale of deployment depending on features such as meeting content archiving that are used in the Lync Server deployment.

4.2.3.2 Lync Server Workloads Storage

Note: The following recommendations ask for a single VHD or physical disk for the VM guest operating system and installed application. We also recommend separate operating system and application (binary) installation virtual hard drives (VHD) are recommended. The decision to implement a separate dedicated operating system VHD may depend on your preferences for your actual virtualized deployment.

Single virtual drive in guest VM , attached as a virtual IDE using a Fixed Disk, or Passthrough Disk. The virtual drive is placed on a dedicated physical spindle, or a dedicated LUN in a SAN. Workloads:

Front End Server

Director

Edge Server

Stand-alone Mediation Server

Stand-alone A/V Conferencing Server

Archiving/Monitoring Server

Survivable Branch server (not applicable to OEM Survivable Branch Appliance)

Two virtual drives in guest VM, attached as virtual IDE (optionally second as virtual SCSI) using Fixed Disk, or Passthrough Disk. The virtual drives are placed on dedicated physical spindles, or a dedicated LUN in a SAN. Workload:

Standard Edition server

4.2.3.3 SQL Server Workloads Storage

All virtual drives are placed on dedicated physical spindles, or a dedicated LUN in a SAN. This is especially critical for the SQL Server storage virtual drives.

SQL Server Back End Server to a Front End pool

Four virtual drives in guest VM. Boot operating system drive attached as a virtual IDE using Fixed Disk. Three drives for SQL Server storage attached using virtual SCSI attached as a Passthrough Disk. However with slightly lower performance requirements, Fixed Disk for SQL Server storage is also acceptable.

The three SQL Server storage drives map the following way: Rtcdyn log, Rtc log, and all other databases and logs.

SQL Server Back End Server for Archiving and Monitoring

17


Three virtual drives in guest VM. Boot operating system drive attached as a virtual IDE using Fixed Disk. Two drives for SQL Server storage attached using virtual SCSI attached as a Passthrough Disk. However with slightly lower performance requirements, Fixed Disk for SQL Server storage is also acceptable.

The two SQL Server storage drives map the following way: Log files, DBs files.

Important: Setting up databases in Lync Server on a virtualized SQL Server requires the use of the Install-CsDatabase cmdlet with the DatabasePath parameter set to explicitly allocate the designated SQL Server storage virtual drives. We do not recommend placing database or log files on the guest operating system drive.

Important: We do not recommend collocating back-end databases and Archiving and Monitoring databases into the same virtual SQL Server workload.

4.2.3.4 File Storage

Two virtual drives in guest VM, with the guest operating system drive attached as virtual IDE using Fixed Disk, and the second drive attached as virtual SCSI using a Passthrough Disk (acceptable Fixed Disk). The second drive is used for the Lync Server file share.

The virtual drives are placed on dedicated physical spindles, or a dedicated LUN in a SAN. Note: For more details and recommendations about storage options, see “Storage options for Windows Server 2008 Hyper-V” at http://go.microsoft.com/fwlink/?LinkId=210858.

4.3 Other VM Guest Configuration Settings

4.3.1 Guest Operating System

Windows Server 2008 R2 is the required guest operating system. Windows Server 2008 SP2 is not supported.

4.3.2 Network adapter

Always use the synthetic VM network adapter in the guest operating system. The use of the legacy network adapter is not supported for a guest operating system running a Lync Server media workload.

Use IP version 4 (IPv4) only in the VM guest and VM host. Disable IPv6 on both the guest and host for improved performance.

4.3.3 Ratio of Physical to Virtual CPUs

Do not oversubscribe physical to virtual CPUs, specifically:

Oversubscription of CPU cores on the host running virtualized Lync Server media workloads is not supported.

Oversubscription of CPU cores on the host is not recommended when running other non-media virtualized Lync Server workloads (such as Director and Archiving and Monitoring Server), or supporting virtualized storage workloads (SQL Server or file server).

Note: Virtual machine reserved allocations are not explicitly necessary. If CPU oversubscription is implemented, consider allocating virtual machine reserves for critical workloads such as the SQL Server Back End Server.

18



4.3.4 Support of VM Image TemplatesLync Server 2010 does not support sysprep, which is applicable also to Microsoft® SQL Server 2008 Express, which is installed on each Lync Server role. This implies that a preconfigured generic Lync Server VM image templates cannot be created.

We recommend the use of custom operating system templates with applicable operating system prerequisites installed depending on the targeted workload, for example an operating system template for Front End Servers containing Internet Information Services (IIS), Windows Media® Format Runtime, Message Queuing (also known as MSMQ).

As a separate VM configuration step, Lync Server roles get deployed and activated after the guest operating system is fully configured.

4.3.5 Disable Virtual DVD/CD Drives in Guest Operating System

If virtual DVD/CD drives are not required in the guest operating system, it is a best practice to disable these drives for better performance.

5 Scale Comparison of Virtualized and Physical Lync Server Roles

5.1 CPU

We recommend an 8 CPU core (Enterprise class CPU) hardware profile for a physical Lync Server. However, Hyper-V supports a maximum of 4 virtualized CPU core. Due to this resource constraint, a virtualized Lync Server role can handle approximately 50% of the load compared to a physical server running Lync Server and using the recommended hardware specifications.

In addition, a 10% virtualization overhead is deducted.

5.2 Memory

Required memory of a virtualized Lync Server 2010 role does not scale down in a linear way by 50%. A recommendation per workload is provided in the table below.

19


5.3 Scale Comparison Table Physical-Virtual

Standard User model with conferencing, PSTN conferencing, and Enterprise Voice

IM&P = IM and Presence

AS = Application Sharing

AV = A/V Conferencing (collocated if Front End Server)

EV = Enterprise Voice (Response Group Service/PSTN access), Mediation Server

Reg = Registrar (subset of IM&P)

AP = Access Proxy (Edge Server)

DP = Data Proxy (Edge Server)

MR = Media Relay (Edge Server)

Note: The Virtual Machine numbers are based on a 4 core CPU Virtual Machine.

Lync Server Role Features Physical Virtual

CPU Memory User CPU Memory User

Enterprise Edition Front End Server

IM&P 8 core 12 GB 20K 4 core 10 GB 12.5K


IM&P, AS, AV, EV 8 core 16 GB 5K 4 core 16 GB 2.5K


IM&P, AS, EV

(8 Front End pool)

8 core 12 GB 10K 4 core 11 GB 5K


IM&P, AS, EV

(single Front End Server in pool)


Standard Edition Front End Server

IM&P 8 core 16 GB 25K 4 core 12 GB 12.5K

Standard Edition Front End Server

IM&P, AS, AV, EV 8 core 16 GB 5K 4 core 16 GB 2.5K

Director Reg 4 core 4 GB 20K 4 core 4 GB 10K

Monitor/Archiving Server


A/V Conferencing Server

AV 8 core 16 GB 20K 4 core 11 GB 10K

Mediation Server* MS 8 core 16 GB 800 concurrent

calls

4 core 10 GB 400 concurrent

calls

Edge Server AP, DP MR 8 core 16 GB 15K 4 core 8 GB 7.5K

20


Survivable Branch Server

Reg, MS 2 core 2 GB 1K 2 core 2 GB 1K

SQL Server Back End Server (FE)


SQL Backend (M/A) 8 core 16 GB 230K 4 core 12 GB 115K

File Server 4 core 4 GB 80K 2 core 3 GB 40K

Table 7 Virtual-physical scale comparison table

* Stand-alone Mediation Server for deployments without Media Bypass

5.3.1 Maximum Conference Size Physical-Virtual

Physical deployment: 250 users

Virtualized deployment: 125 users (4 core CPU Virtual Machine)

5.3.2 Stand-alone A/V Conferencing Server Recommendation Physical-Virtual

Recommendation for deploying a dedicate stand-alone A/V Conferencing Server or pool separate from an Enterprise Edition Front End pool:

Physical Lync Server deployment: 10K or more users

Virtual Lync Server deployment: 5K or more users (4 core CPU Virtual Machine)

6 Sample Lync Server Virtualized Topologies

Note: The diagrams below are examples, and it is not required to host a specific combination of VM workload on any given VM. The example VM host specification here is a 16 core server, with 64 GB of memory.

6.1 Virtual Enterprise Edition Deployment A

All virtualized topology with eight Front End Servers, four A/V Conferencing Servers, one SQL Server Back End Server, one SQL Server for archiving and monitoring, one Archiving and Monitoring Server (collocated), one File Server, and – on a separate VM host in the perimeter network (also known as DMZ) – two Edge Servers.

Scale: 40K users, default usage profile, all modalities

21


Figure 4 Virtual Enterprise Edition deployment A

6.2 Virtual Enterprise Edition Deployment B

All virtualized topology with four Front End Servers, three A/V Conferencing Servers, one SQL Server Back End Server, one SQL Server for archiving and monitoring, one Archiving and Monitoring Server (collocated), one File Server, and (with VLAN into the perimeter network) single Edge Server.


22


Figure 5 Virtual Enterprise Edition deployment B

23


6.3 Mix Virtual/Physical Enterprise Edition Deployment

Mixed virtualized/physical topology with eight Front End Servers, four A/V Conferencing Servers, one SQL Server Back End Server (physical Back End and Archiving and Monitoring Store in same SQL Server instance), one Archiving and Monitoring Server (collocated), one File Server (physical), and two Edge Servers (on physical server). Survivable Branch Server virtualized (example for single site), and other workloads are running on the VM host in the branch.


Figure 6 Mixed virtual-physical Enterprise Edition deployment

24


6.4 Virtual Standard Edition Deployment

All virtual Standard Edition deployment, single Standard Edition server, with Archiving and Monitoring, SQL Server (Archiving and Monitoring), Edge Server (VLAN configuration to perimeter network)


Figure 7 Virtual Standard Edition deployment

25


7 Test Results

Virtual Machine Queue (VMQ) should be enabled wherever supported by network adapters. VMQ is a virtualization technology for the efficient transfer of network traffic to a virtualized operating system. VMQ allows the VMs to filter the queue of packets within the network adapternetwork adapter, resulting in improved efficiency of network traffic. VMQ requires support in the hypervisor, as well as using VMQ-enabled network adapternetwork adapters (limited to Intel and Broadcom based network adapternetwork adapters). You can enable this for each virtual machine using the hypervisor’s management console, assuming that the host’s network adapternetwork adapter supports this. For details about enabling VMQ, see Appendix A.

Ensure that the latest network adapternetwork adapter driver from the hardware manufacturer is installed.

Increase the network adapternetwork adapter’s send/receive buffers to at least 1024 on any network adapternetwork adapter dedicated to virtual machines. Increasing the send/receive buffer allows the operating system greater time to process the network queue, and helps to avoid packet loss. For details about how to do this, see Appendix B.

26


8 Measuring Performance on Virtual Lync Server 2010The testing described in the previous section required interpreting the Lync Server user model, and understanding how to measure the performance of Lync Server. This section describes the user model and how it was used to measure performance on virtual machines that are running Lync Server.

Additionally, this section describes the key performance indicators (KPIs) to monitor for virtual Lync Server performance.

8.1 Performance Indicators

For details about specific performance counters and where to observe them, see Appendix D: Performance Counters.

8.1.1 Processor and Percentage of Processor Time

In general, all processors and percentage of total processor time should be measured on each server.

Short CPU spikes of 70% or more for less than a few minutes is acceptable, however, the CPU should typically be operating at less than 70% during peak load. If it is consistently higher, there could be more users connected than usual, or some other change resulting in higher load. This indicates that the process is overloaded, and more instances of the server are needed to handle that load.

8.1.2 Network Interface

The network adapternetwork adapters should be operating at no more than 80% capacity.

8.1.3 Memory

Pages/second indicates the total pressure on the server’s available memory.

8.1.4 Databases Queue Latency: The amount of time a request spends in the queue to the Back End Server.

If for any reason resources (such as disk, memory, network, and processor) on the Back End Server are too high, the queue latency will also be high. Queue latency can also be high if there is too much network latency between the Front End Server and the Back End Server.

Sproc Latency: The amount of time it takes the Back End Server to process a request.If both Sproc latency and Queue latency counters are high, then the most likely bottleneck is the SQL Server back-end database. Check for disk IO contention first (see disk). SQL Server database active transactions are a good indicator of which database is having more contention than others.

Page Life Expectancy: The server needs more RAM if this counter is a value fewer than 3600, which indicates memory pressure by the system. Note: there should be few ‘reads’ on the database drive if there’s sufficient memory because SQL Server should only be writing to the database.

8.1.5 SIP Average Holding Time for Incoming Messages: If this counter is more than 10 seconds (12

seconds maximum), then the server goes into throttling mode and only allows one outstanding

27


transaction per client connection. If this value goes down to 3 seconds, then the server allows up to 20 outstanding transactions per client connection.

Flow Controlled Connections: Whenever the number of transactions on a specific connection exceeds the above threshold, the connection is marked as flow controlled and the server doesn’t post any receives on it, and this counter is incremented.

Local 503 Responses/sec: The HTTP 503 return code corresponds to server unavailable. There shouldn’t be any of these responses on a healthy server in a steady state. There may be some during ramp up, after a server has just been brought back online, but once all users reconnect and the server reaches a stable state there should no longer be any of these responses.

Local 504 Responses/sec: A few of these responses to clients are to be expected (for example, when clients abruptly disconnect), but this counter mostly indicates connectivity issues with other servers. This can indicate connection failures or delays connecting to remote servers.

Sends Outstanding: If the 504 performance counter indicates that 504s are being received, then this is a good counter to check. This value indicates the number of requests and responses that are queued outbound: if this counter is high then the problem is probably not on this server. This counter can also be high if there are network latency issues.

28


Appendix A: How to Enable VMQ on Intel Network AdapterNote: If you are using a network adapter teaming with the Intel network adapter, ensure that you have Intel v15.0 (or higher driver available). Earlier versions do not support VMQ on a teamed network adapter.

In a command prompt window, run the following to add the key to the host machine’s registry:reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VMSMP\Parameters\BelowTenGigVmqEnabled /t REG_DWORD /d 1 /f

Then, enable VMQ on the driver:1. In Control Panel, open the Device Manager and browse to Network Adapters.2. Right-click the applicable network connection, click Properties, and then click the Advanced

tab.3. In the Settings list, click Virtual Machine Queues, and then set the value to Enabled.

Figure 8 Advanced tab of Network Adaptor Properties, showing Virtual Machine Queue setting

29


Appendix B: How to Increase Send/Receive Buffers on Intel Network Adapter

1. In Control Panel, open Device Manager and browse to Network Adapters.

2. Right-click the applicable network connection, click Properties, and then click the Advanced tab.

3. In the Property list, click Receive Buffers, and increase the value to at least 1024.

4. In the Property list, click Transmit Buffers, and increase the value to at least 1024.

5. Depeding on the driver version used, Receive and Transmit Buffers may not be visible here, and rather “Performance Options” may be listed. If this is the case, open it and browse to the settings for Transmit and Receive Buffers.

6. Click OK to save these settings.

Figure 9 Advanced tab of Network Adapter properties, showing Receive Buffers property.

30


Appendix C: Changing the Bandwidth Limit for Application SharingYou can change the upper bandwidth limit used by Application Sharing server by using the Set-CsConferencingPolicy cmdlet in Lync Server Management Shell. 150 Mbps is the recommended limit; after this point the virtual machines were observed to become CPU bound.

Run the following example cmdlet on the virtual Standard Edition server to change the Application Sharing bandwidth:Set-CsConferencingConfiguration -Identity global –MaxBandwidthPerAppSharingServiceMb 150

31


Appendix D: Performance CountersIn order to monitor performance in a virtual Lync Server site, the performance counters described in this section should be observed. For details about how to create performance baselines, see the second part of this section.

1 Lync Server 2010 Performance Counter ReferenceFor reference, the performance counter information in this section is separated by the Lync Server role as follows:

All Servers Front End Server Edge Server Back End Server Mediation Server A/V Conferencing Server

1.1 All Servers

Monitor the following performance on all virtual Lync Server system:Performance Counter DetailsProcess(*)\%Processor Time Monitor and collect (for Lync Server processes)Process(*)\%Privileged Time Monitor and collect (for Lync Server processes)

Process(*)\%Private Bytes Monitor and collect (for Lync Server processes)

Process(*)\%Virtual Bytes Monitor and collect (for Lync Server processes)

Processor\% Processor Time (Total) It’s OK to see short CPU spikes at 70% or more – less than a few minutes, but the CPU should normally operate at less than 70% during peak load. If it is consistently higher, then determine if there are more users connected than usual or if there has been some other change that may result in higher load. Note this counter is representing the virtualized CPU as the guest operating system sees it.

LS:SIP – 07 – Load Management; SIP – 000 – Average Holding Time For Incoming Messages

This should usually be less than 1 second on average but it’s normal to see short spikes of up to 3 seconds. The server will throttle new incoming messages after going above the high watermark and until the number of messages falls below the low watermark. Check Outgoing Queue Delay for delays sending messages to other servers or clients that could be causing messages to be accumulated in the server. The server will drop client connections if it’s in a throttle state and messages stay in the outgoing queue for more than 32 seconds.The server starts rejecting new connections when average holding time

LS:SIP – 01 – Peers\SIP – 020 – Average

32


is greater than overload time of 15 seconds.LS:USrv – 21 – Https Transport; USrv – 002 – Number of failed connection attempts / Sec

Monitor and collect

1.2 Front End Servers

Monitor the following Lync Server modalities on Front End Servers:Performance Counter Details\LC:AsMcu – 00 – AsMcu Conferences\ASMCU – 010 – Connected Users Compare these

counters’ maximum to the published conferencing numbers in this document.

\LC:AsMcu – 00 – AsMcu Conferences\ASMCU – 010 – Active Conferences\LC:ImMcu – 00 – ImMcu Conferences\IMMCU – 010 – Connected Users\LC:ImMcu – 00 – ImMcu Conferences\IMMCU – 010 – Active Conferences\LC:RGS – 01 – response group service call control\RGS – 009 – Current active calls\LC:SIP – 01 – Peers(Clients)\SIP – 000 – Connections Active\LC:DATAMCU – 00 – DataMCU Conferences\DATAMCU – 003 – Active Conferences\LC:USrv – 06 – Rich Presence Service SQL Calls\ISrv – 005 – Publications/Sec\LC:MediationServer – 00 – Outbound Calls(_Total)\- 000 Current\LC:MediationServer – 01 – Inbound Calls(_Total)\- 000 Current

Monitoring the individual processes on Front End Servers:Performance Counter DetailsProcessor\(_Total)\%ProcessorTime Monitor and collect\Processor\(ASMCUSvc)\%ProcessorTime\Processor\(IMMCUSvc)\%ProcessorTime\Processor\(MediationServerSvc)\%ProcessorTime\Processor\(OcsAppServerHost#1)\%ProcessorTime\Processor\(OcsAppServerHost#2)\%ProcessorTime\Processor\(OcsAppServerHost#3)\%ProcessorTime\Processor\(OcsAppServerHost#4)\%ProcessorTime\Processor\(OcsAppServerHost#5)\%ProcessorTime\Processor\(OcsAppServerHost#6)\%ProcessorTime\Processor\(RTCSrv)\%ProcessorTime\Processor\(RtcHost)\%ProcessorTime

Monitor following SIP/IMMCU/Database Store performance counters on all Enterprise Edition Front End Servers:Performance Counter DetailsLS:USrv – 01 - DB Store; USrv – 002 - Queue Latency (msec)

Queue Latency: the time a request spends in the queue to the Back End Server. The server will start dropping requests when the database queue latency is >32 seconds. Sporadic spikes of a few seconds are considered normal,

33


however the average latency should be < 100msec.Server health decreases as latency increases to 32 seconds, after which server throttling begins.

LS:USrv – 01 - DB Store; USrv – 004 - Sproc Latency (msec)

Note: If for any reason (such as disk, memory, network, or processor) on the Back End Server is too high, the queue latency counter will be high. Queue latency can also be high if there is too much network latency between the Front End Server and the Back End Server. If both Sproc Latency and Queue Latency counters are high, then most likely the bottleneck is the SQL Server back-end database. Check for disk IO contention first (see Physical Disk perf counters). SQL Database active transactions are a good indicator of which database is having more contention than the others.

LS:USrv – 01 - DB Store; USrv – 020 – Throttled requests/sec

Rarely, requests are dropped from the queue due to database queue contention.

LS:Urv – 13 – Endpoint Cache;USrv – 001 – Active Registered EndpointsLS:SIP – 07 – Load Management;SIP – 000 – Average Holding Time For Incoming MessagesLS:SIP – 07 – Load Management;SIP – 004 –Incoming Messages Held Above High WatermarkLS:SIP – 07 – Load Management;SIP – 005 –Incoming Messages Held Above High Overload WatermarkLS:SIP – 07 – Load Management;SIP – 006 –Incoming Messages Timed outMSSQL$RTCLOCAL: Buffer Manager - Page life expectancy

The server could need more RAM if this counter is < 3600 which indicates memory pressure by the system. Additionally, there are few to no ‘reads’ on the database drive if there’s enough memory, because SQL Server should only be writing to the database.

LS:SIP - 07 - Load Management; SIP - 000 - Average Holding Time For Incoming Messages LS:SIP – 04 – Responses; SIP – 053 - Local 500 Responses/sec

Indicates that there’s some server component that is not functioning correctly.

LS:SIP – 04 – Responses; SIP - 055 - Local 503 Responses/sec

The 503 code corresponds to server unavailable. There should not be any 503s on a healthy server at a steady state. During ramp up, after a server has just been brought back online, there may be some 503s. However, as all users get back in and the server returns to a stable state, there should no longer be any 503s.

LS:SIP – 04 – Responses; SIP – 057 - Local 504 Responses/sec

A few 504 responses to clients (for clients disconnecting abruptly) is to be expected, but this counter mainly indicates connectivity issues with other servers. It can indicate connection failures or delays connecting to remote servers.

LS:SIP – 01 – Peers; SIP - 017 - Sends Outstanding (*) If there are 504 responses, this is a good counter to check. It indicates the number of requests and responses that are queued outbound, which means that if this counter is high then the problem is probably not on this server. This counter can be high if there are network latency issues

34


LS:ImMcu – 02 – MCU Health And Performance; IMMCU – 005 – MCU Health State

The current health of the Instant Messaging “MCU”. 0 = Normal 1 = Loaded2 = Full3 = Unavailable.

LS:USrv– 23 – Conference Focus Factory; USrv – 012 – Add Conference requests succeeded

Compare to total add conference requests to determine conference allocation failures.

LS:USrv– 24 – Conference Control; USrv – 018 – Local C3P failure responses

These counters indicate conferencing related failures.

LS:USrv– 26 – Conference Mcu Allocator; USrv – 009 – Factory Unreachable FailuresLS:USrv– 26 – Conference Mcu Allocator; USrv – 010 – Factory Calls Timed-OutLS:USrv– 26 – Conference Mcu Allocator; USrv – 016 – Create Conference Mcu Unreachable FailuresLS:USrv– 26 – Conference Mcu Allocator; USrv – 017 – Create Conference Requests Timed-Out

Monitoring the following Performance Counters for the Web Servers component (applicable to Front End Server and Director role)Performance Counter Details

LS:DATAMCU – 00 – DataMCU Conferences; DATAMCU – 003 – Active Conferences ----LS:DATAMCU – 00 – DataMCU Conferences; DATAMCU – 025 – Number of Data MCU users in any role --LS:DATAMCU – 04 – MCU Health and Performance; DATAMCU – 005 – MCU Health State

The current health of the MCU. 0 = Normal. 1 = Loaded. 2 = Full. 3 = Unavailable.

LS:DATAMCU – 04 – MCU Health and Performance; DATAMCU – 006 – MCU Draining State

The current draining status of the MCU. 0 = Not requesting to drain. 1 = Requesting to drain. 2 = Draining.

Monitor the following performance For application sharing (ASMCU), observe the following:Performance Counter DetailsLS:AsMCU – 02 – CCCP Processing ---- Monitor and collectLS:AsMCU – 017 – CCCP Messages Retried/Sec ----LS:AsMCU – 019 – Current CCCP Message Queue Size ----LS:AsMCU – 029 – Number of add conference requests failed ----LS:AsMCU – 030 – Number of add user requests failed ----LS:AsMCU – 00 – AsMcu Conferences\ASMCU – 000 – Active ConferencesLS:AsMCU – 00 – AsMcu Conferences\ASMCU – 003 – Connected Users

35


1.3 Edge Server

Monitor the following performance counters on Edge Server (Access Edge SIP component):Performance Counter Details\LC:SIP – 09 – Access Edge\SIP – 001 – External Messages/sec with Internally Supported Domain

Monitor and collect

\LC:SIP – 09 – Access Edge\SIP – 003 – External Messages/sec Received With Allowed Partner Server Domain\LC:SIP – 09 – Access Edge\SIP – 001 – External Messages/sec Received With IM Service Provider Domain\LC:SIP – 09 – Access Edge\SIP – 001 – External Messages/sec Received With a Configured Allowed Domain

Monitor the A/V Conferencing Edge component (audio/video and application sharing):LS:A/V Edge – 00 UDP Counters:

Performance Counter Details

LS:A/V Edge - 001 - Active Relay Sessions – Authenticated

LS:A/V Edge - 002 - Active Relay Sessions - Allocated Port

LS:A/V Edge - 003 - Active Relay Sessions – Data

LS:A/V Edge - 004 - Allocated Port Pool Count The value must be > 0. If it reaches 0 there is a resource issue.

LS:A/V Edge - 005 - Allocated Port Pool Miss Count

LS:A/V Edge - 006 - Allocate Requests/sec

LS:A/V Edge - 009 - Allocate Requests Exceeding Port Limit If the value is > 0, this could indicate an attempt to misuse the port.

LS:A/V Edge - 012 - Alternate Server Redirects

LS:A/V Edge – 019 – Session Idle Timeouts/sec

LS:A/V Edge - 021 - Packets Received/sec

LS:A/V Edge - 022 - Packets Sent/sec

LS:A/V Edge - 025 - Average Data Packet Latency (milliseconds)

LS:A/V Edge - 030 - Packets Dropped/sec This error occurs when an unexpectedly high rate of User Datagram Protocol (UDP) packets is received at the Media Relay (A/V Edge server) causing some packets to be discarded. This could be the result of system overload or an indication of an attempt to misuse the MR.To resolve this, check that the profile of network traffic to the MR is in line with expected usage. If the traffic exceeds 250 Mbps per interface then increase the Receive and Transmit buffer size on the associated network adapternetwork adapters to three times the default values.

36


If the cause is a general system overload, then increase capacity of the deployed MR function. A network level trace can be used to determine if there is an unusual amount of traffic originating from a single source. If the situation persists, then enable tracing to check the network source of sessions exceeding the bandwidth limits to allow further troubleshooting of the cause.

LS: A/V Edge – 01 – TCP Counters:


LS:A/V Edge - 001 - Active Relay Sessions – AuthenticatedLS:A/V Edge - 002 - Active Relay Sessions - Allocated PortLS:A/V Edge - 003 - Active Relay Sessions – DataLS:A/V Edge - 004 - Allocated Port Pool Count The value must be > 0. If it reaches 0 there is a

resource issue.LS:A/V Edge - 005 - Allocated Port Pool Miss CountLS:A/V Edge - 006 - Allocate Requests/secLS:A/V Edge - 009 - Allocate Requests Exceeding Port Limit If the value is > 0, this could indicate an

attempt to misuse the port.LS:A/V Edge - 012 - Alternate Server RedirectsLS:A/V Edge – 019 – Session Idle Timeouts/secLS:A/V Edge - 021 - Packets Received/secLS:A/V Edge - 022 - Packets Sent/secLS:A/V Edge - 025 - Average Data Packet Latency (milliseconds)LS:A/V Edge - 030 - Packets Dropped/sec This error occurs when an unexpectedly high

rate of TCP packets is received at the MR causing some packets to be discarded. This could be the result of system overload or an indication of an attempt to misuse the MR.To resolve this, check that the profile of network traffic to the MR is in line with expected usage. If the traffic exceeds 250 Mbps per interface then increase the Receive and Transmit buffer size on the associate network adapter to three times the default values. If the cause is general system overload then increase capacity of the deployed MR function. A network level trace can be used to determine if there is unusual amount of traffic originating from a single source. If the situation persists then enable tracing to check the network source of sessions exceeding the

37


bandwidth limits to allow further troubleshooting of the cause.

Monitor the following on the HTTP relay (Web Server component with Reach client):Performance Counter DetailsLS:WebRelay – 00 – Reach Web Relay Server\* Monitor and collect

1.4 Back End Server

Memory (not applicable if guest VM has dynamic memory enabled)Performance Counter Details% Committed Byes in Use Monitor and collectAvailable MbytesCached Bytes/[peak]

Network Interface:

Performance Counter DetailsBytes Total/sec <All instances> Monitor and collect

Physical Disk: Performance Counter Details% Idle Time (*) A consistently small percentage of idle time is usually an

indication of poor disk performance.Ave. Disk Queue Length (*) Disk Queue Length per drive should be no more than double the

number of spindles backing up that drive.Avg. Disk Sec/Read(*) Any average numbers above 1.5 msec usually indicate poor disk

performance.Avg. Disk Sec/Write(*)

Process:Performance Counter DetailsProcess(*)\%Processor Time Monitor and collectProcess(*)\%Privileged TimeProcess(*)\%Private BytesProcess(*)\%Virtual BytesProcessor Information:

% Interrupt Time (_Total)% Processor Time (_Total)

Short CPU spikes of around 80% or more for less than a few minutes is to be expected, but the CPU should normally operate at less than 80% during peak load. If it is consistently higher, then determine if there are more users connected than usual or if there has been some other change that may result in higher load.

Database:Performance Counter DetailsSQLServer:User Settable\User Counter 1 Approximate number of active endpoints in the pool.MSSQL$RTC:Databases(rtc)\Active Transactions

A high number of active transactions usually indicates database transaction contention.

38


MSSQL$RTC:Databases(rtcdyn)\Active Transactions

Monitor and collect

MSSQL$RTC:Databases(tempdb)\Active Transactions

1.5 Mediation Server

The following performance counters should be used to measure the performance of the Mediation Server:Performance Counter DetailsLS:MediationServer - 00 - Outbound Calls\- 000 – Current

The total number of active calls going through the Mediation Server.

LS:MediationServer - 00 - Outbound Calls\- 006 – Active media bypass calls

Total number of active calls going through Mediation Server that are in Media Bypass mode. Calls using Media Bypass use significantly fewer Mediation Server resources because the media is not flowing through Mediation Server.

LS:MediationServer – 04 – Global Counters\-000 Current Audio channels with PSM quality reporting

Total number of active channels that are having Phase Shift Modulation (PSM)quality reported. Calculating PSM quality has a processing overhead so this should be taken into account when measuring performance.

LS:MediationServer - 01 - Inbound Calls\- 000 – Current

Monitor and collect

LS:MediationServer - 00 - Inbound Calls\- 006 – Active media bypass calls

39


The following performance counters can be useful when troubleshooting the Mediation Server component:Performance Counter DetailsLS:MediationServer - 02 - Media Relay\- 001 - Media Connectivity Check Failure

The number of calls where media connectivity between Mediation Server and the remote endpoints could not be established.

LS:MediationServer - 03 - Health Indices\- 000 - Load Call Failure Index

Scaled index between zero and 100 that is related to all call failures due to Global Health Index as heavy load.

CS:MediationServer - 04 - Global Counters\- 001 - Total failed calls caused by unexpected interaction from the Proxy ----

The number of calls that failed because of an unexpected response from the Front End Server.

CS:MediationServer - 05 - Global Per Gateway Counters\- 000 - Total failed calls caused by unexpected interaction from a gateway

The number of calls that failed because of an unexpected response from a gateway peer.

1.6 Audio/Video Conferencing Server

Monitor the following Performance Counter with applicable threshold levels (implement in System Center Operations Manager Lync Server Management)


LS:MEDIA - 01 – Planning\MEDIA - 009 - Number of occasions conference processing is delayed significantly\ AVMCUSvc.exe

Error Threshold = 30 Experiencing transient audio quality issues due to CPU load

LS:SipEps - 02 - SipEps Connections\SipEps - 010 – NumberOfDNSResolutionFailures\*avmcusvc*

Error Threshold = 1 (in 4 consecutive)

The events in the Lync Server event log reflect media stack health:Event ID Details32013 Audio Video Conferencing Server health monitoring has detected that Audio Video Conferencing Server is running at a moderate capacity (Warning or Error)

If the A/V Conferencing Server reaches these points then subsequent operations to add new conferences and new users will be rejected until the health returns to normal state.

1.7 Establishing Performance Baselines

In order to establish a performance baseline, use the following counters

1.7.1 Resource Utilization Baseline Tests

Processor; % Processor Time (_Total)

Process; % Processor Time (RtcSrv)

Process; % Processor Time (IMMcuSvc)

Memory; Pages/sec ---

Network Interface; Bytes Total/sec ([your network adapter])

40


1.7.2 User Load Baseline Tests

LS:SIP – 01 - Peers; SIP - 028 - Incoming Requests/sec (_Total)

LS:SIP - 01 – Peers; SIP – 001 – TLS Connections Active (_Total)

LS:SIP – 01 – Peers; SIP – 000 – Connections Active (_Total)

LS:SIP – 02 – Protocol; SIP - 001 - Incoming Messages/sec ----

LS:ImMcu – 00 - IMMcuSvc Conferences; IMMCU – 000 - Active Conferences ----

LS:ImMcu – 00 - IMMcuSvc Conferences; IMMCU – 001 – Connected Users ----

LS:USrv – 00 – DBStore; Usrv – 002 – Queue Latency (msec)

LS:USrv – 00 – DBStore; Usrv – 004 – Sproc Latency (msec)

1.7.3 Multipoint Conferencing Unit (MCU) Resource Utilization Baseline Tests: (Monitor on All MCUs)

Processor; % Processor Time (_Total)

Network Interface; Bytes Total/sec ([your network adapter])

Memory; Pages/sec (---)

Process; % Processor Time (ASMCUSvc)

Process; % Processor Time (AVMCUSvc)

Process; % Processor Time (DataMCUSvc)

Process; % Processor Time (MeetingMCUSvc)

Process; Private Bytes (ASMCUSvc) ([peak])

Process; Private Bytes (AVMCUSvc) ([peak])

Process; Private Bytes (DataMCUSvc) ([peak])

Process; Private Bytes (MeetingMCUSvc) ([peak])

41

Documents

Server Virtualization in Microsoft Lync Server 2010