Failover ClusteringMicrosoft Cluster Service (MSCS) is available for installation on Windows 2000 Advanced Server, Windows2000 Datacenter Server, and Windows NT Enterprise Edition with Service Pack 5 (although this is notrecommended). MSCS allows applications, such as SQL Server 2000 Enterprise Edition, to take advantage ofhigh-availability failover functionality. A failover means that if one server, or node, fails, the applications onanother server in the cluster will start up and take over for the failed node. There are two types of failovers,planned and those that occur as a result of a server hardware or software problem. Only applications that arecluster-aware can utilize the MSCS failover capabilities. Cluster-aware applications, such as SQL Server, willremain available even if one node encounters a failure. Windows 2000 Advanced Server supports two-nodeclustering, and Windows 2000 Datacenter supports four-node clustering.

MSCS achieves high availability by using a shared disk array. For SQL Server clustering, this shared diskarray contains the database files (.mdf, .ndf, .ldf), along with backups, and other optional components.The shared disk can be a regular external SCSI disk or disk array, or a Storage Area Network (SAN) array.SAN arrays are sold by hardware and software vendors, and allow high-speed data transfer to external diskarrays via a fiber channel connection (and network card). Software RAID is not supported; only hardwareRAID configurations are allowed. File encryption is also unsupported for the shared disk(s).

Binary installation files and executables are installed locally onboth nodes of a two-node cluster, and are not placed on the shareddisk. The SQL Server and SQL Server Agent services are alsoinstalled on each node of the cluster. For a SQL Server 2000virtual server, the same service is installed on both nodes, butonly one side is active at any time. Virtual server is another way ofsaying that the SQL Server instance is on a failover cluster. Clientsand applications connect to the virtual server as they would for anonclustered default or named instance of SQL Server. The SQLServer 2000 virtual server can exist or be controlled by eithernode within a cluster (but only one at a time):

Chapter 10

550

If a failure occurs on the active node, the second node will 'go live'; its services will start up and take overcontrol of the files on the shared disk array for the SQL Server 2000 virtual server:

In a two-node cluster, two servers are connected using a private network (heartbeat) connection. The privatenetwork gives the two nodes a fast path for checking each other's status. Network connectivity for end users isestablished via a public IP address. End users connect to the SQL Server 2000 virtual server using the virtualserver NetBIOS name. This name and TCP/IP address is defined prior to SQL Server installation, and isspecified during installation.

A SQL Server 2000 virtual server is defined by a cluster group, which in turn is made up of separateresources. Resources included in a SQL Server 2000 virtual server group include a shared disk drive or drives,the virtual server name, the virtual server IP address, the SQL Server and SQL Server agent services and,optionally, the SQL Server Fulltext service. This group is treated as a single unit where, in the event of a nodefailure, all resources must be started or engaged on the failover node. Resources and groups are managed inCluster Administrator, which will be reviewed later in this chapter.

A two-node cluster needs the following names and IP addresses:

Type Description

Physical server name and TCP/IP address for firstnode

Just like a normal server requirement, you havea name and associated IP address.

Physical server name and TCP/IP address forsecond node

…

Cluster name and TCP/IP address The cluster name is a virtual name and IPaddress used for adding MSCS, and isreferenced in Cluster Administrator (more onthis later on).

Heartbeat name and TCP/IP address for Node 1 The heartbeat or private network connectionbetween the two nodes for checking status andavailability. These should be on the samesubnet.

Failover Clustering

551

Type Description

Heartbeat name and TCP/IP address for Node 2 …

SQL Server 2000 virtual server name and TCP/IPaddress

This is the name that SQL Server users will useto access the clustered instance. This instancecan exist on either Node 1 or Node 2. Usersshould treat this instance like any regular SQLServer instance. If you are installing just oneSQL Server instance on the cluster, this is anactive/passive setup.

Second SQL Server 2000 virtual server name andTCP/IP address

If using two instances of SQL Server on onecluster, this is called an active/active setup.

Failover clustering with SQL Server 2000 Enterprise Edition allows an active/passive, or active/active setup.Active/passive, also called single instance, means the failover cluster has one node that remains unused untila failover occurs. You can have a SQL Server named instance called BookRepo\INS1 that is managed byNode1 by default, but can fail over to Node2:

Active/active configuration, also called multi-instance, makes use of both nodes in a cluster. You can run twoinstances of SQL Server on the cluster. For example, Node1 can run BookRepo\INS1 and Node2 can runBookRepo\INS2. Both instances are separate installations of SQL Server and can fail over to each other'snode if one node fails. The advantage of active/active is that you have twice the SQL Server instances to workwith, and the hardware does not go to waste:

Chapter 10

552

In the event of a failure in Node2, services for both instances of SQL Server would be running on Node1,pointing to the associated shared disk drives in order to keep both instances running.

Failover Clustering

553

It is important to understand that MSCS allows high availability, but not total fault-tolerance. If one nodefails with an existing SQL Server 2000 virtual server, that virtual server will restart and be controlled fromthe second node. Depending on how large your instances are, and the recovery time for your databases,failover can take anywhere from 15 seconds to several minutes. During a failover, any users currentlyconnected will be disconnected as if the server were rebooted.

Also note that failover clustering is not the same as network load balancing (NLB). Workload is notdistributed between the two nodes for one SQL Server instance. NLB is a Microsoft Windows product thatbalances incoming Internet Protocol traffic across multiple nodes, acting as one logical application, andallowing the application to scale out up to 32 nodes. NLB is used primarily for Web and media servers, as wellas Terminal Services. Failover Clustering (the cluster service) on the other hand, is used primarily fordatabase, file, print, and messaging servers.

Nor is a failover cluster a substitute for a disaster recovery plan. You must still produce a disaster recoveryplan, in order to get mission-critical applications back online as quickly as possible (see Chapter 6).

Cluster Meta DataThe Cluster Group manages failover and keeps track of groups and resources. This group uses a shared drivecalled the quorum drive, which manages the cluster and logging:

Do not use this quorum drive to install files on your SQL Server instances. The cluster group usually consistsof the cluster IP address resource, cluster name resource, quorum drive resource, time service resource, andMS DTC service resource. Just like a SQL Server group, the Cluster Group is only managed by one node at atime, but can failover to the second node in the event of a problem.

Cluster AdministratorThroughout this chapter, you will be using the Cluster Administrator. Cluster Administrator can be launchedon either of the cluster nodes, or from your client.

If you are familiar with previous versions of failover clustering for SQL Server, please note thatIf you are familiar with previous versions of failover clustering for SQL Server, please note thatIf you are familiar with previous versions of failover clustering for SQL Server, please note thatIf you are familiar with previous versions of failover clustering for SQL Server, please note thatas an enhancement from SQL Server 2000 onwards, you can now use SQL Server Serviceas an enhancement from SQL Server 2000 onwards, you can now use SQL Server Serviceas an enhancement from SQL Server 2000 onwards, you can now use SQL Server Serviceas an enhancement from SQL Server 2000 onwards, you can now use SQL Server ServiceManager or SQL Server Enterprise Manager to start and stop SQL Server, without having to useManager or SQL Server Enterprise Manager to start and stop SQL Server, without having to useManager or SQL Server Enterprise Manager to start and stop SQL Server, without having to useManager or SQL Server Enterprise Manager to start and stop SQL Server, without having to useCluster Administrator to start and stop SQL Server services.Cluster Administrator to start and stop SQL Server services.Cluster Administrator to start and stop SQL Server services.Cluster Administrator to start and stop SQL Server services.

If you have not installed the Administrative Tools from the Windows 2000 CD, you must do so in order to runCluster Administrator from your client. To install these utilities, run AdminPak.msi from the I386 folder onthe Windows 2000 Advanced Server CD.

To launch Cluster Administrator, go to Start | Programs | Administrative Tools | Cluster Administrator. Afaster method is to go to Start |Run, type in cluadmin and OK.

Chapter 10

554

Cluster Administrator allows you to:

Add new groups (groups are logical groupings of resources).

Add resources to groups (resources being disk drives, IP addresses, network names, services, andmore…)

Fail a group to another node. This means that the group and the resources it contains are taken offlineon the existing node, and brought online on the other node.

Take a group offline (if you need to add new resources or configure group properties).

Configure resource properties and dependencies (resource dependencies help make sure that certainresources are necessary for others to run).

Monitor the status of the nodes and resources (red X marks and yellow caution flags show issues withentities in Cluster Administrator).

This chapter will explain how to use Cluster Administrator in the context of installing and configuring a SQLServer 2000 Virtual Server. This chapter also assumes that MSCS has already been installed and configured.

Pre-Installation Checklist for SQL ServerFailover Clustering

SQL Server 2000 Enterprise Edition is required.

For active/passive, you need to install SQL Server once. For active/active, you install SQL Servertwice, once for each virtual server name. Make sure your name and TCP/IP addresses are reserved foryour virtual server name(s).

Verify that you can ping each of the heartbeat DNS names for the cluster, both for the physical nodenames, and the cluster name. If you get a response problem, make sure that the nodes are running andIP addresses have been configured properly.

Make sure that the shared disk drive letter mappings are the same on both nodes of the cluster.

Make sure that the shared disk drive(s) are accessible.

Make sure your hardware is listed on Microsoft's Hardware Compatibility List (HCL). The fact that youcan get it to work does not mean that Microsoft will support you if you run into problems. Treat thiswarning seriously for your production environments.

Make sure all operating system service packs and patches are installed first.

Make sure NetBIOS is disabled for your private (heartbeat) network properties. Go to Start | Settings |Network | Dialup Connections. Double-click the private network connection. Click Properties. SelectTCP/IP and select Properties. Click the Advanced button. On the WINS tab, make sure NetBIOS isdisabled.

Make sure your network cards being used for both public and private connections are not set to autodetect. Go to Start | Settings | Network | Dialup Connections. Double-click the network connection(make sure to do this for each network connection used). Click Properties. Click Configure. In theAdvanced tab, click Advanced, and select Link Speed & Duplex. Make sure this is not set to Autodetect, but rather to the speed capabilities for the card.

Determine ahead of time the naming convention for your virtual SQL Server name(s) and clustername. With two physical names, two private heartbeat names, one cluster name, and one or morevirtual SQL Server names, keeping track can get a little confusing!

Failover Clustering

555

Do not install a failover cluster on a domain controller. Putting SQL Server instances on a domaincontroller is generally not recommended for nonclustered implementations in general. You mayencounter performance contention if the domain controller is particularly busy, or security issues ifyou require a domain account for running the SQL Server and SQL Server Agent service accounts.

10.1 How to… Install a SQL Server 2000Virtual Server

Prior to installing the SQL Server 2000 instance, you must install a cluster aware version of the MicrosoftDistributed Transaction Coordinator (MS DTC) service.

Open a command line window and type comclust. Press Enter. The MS DTC service will now beinstalled as 'cluster aware':

Repeat step 1 on the second node.

Check to make sure that the MS DTC service was added to the Cluster Group on your cluster. Go toStart | Run | and type cluadmin.

Expand the Group folder and click the Cluster Group (this may have been named something else byyour administrator after the MSCS installation, but this is the group with the quorum disk resources,cluster name, and IP address). Make sure MS DTC is in the group:

While in Cluster Administrator, verify that the node from which you are installing SQL Servercurrently controls the shared disk(s) you plan to use for the SQL Server instance. If not, right-click thegroup containing the disk resources and select Move Group; this will move control of the group to theopposite node.

Insert the SQL Server 2000 CD ROM on the installation node. Installing a SQL Server instance forclustering from a network drive can cause difficulties and does not always work. If the main splashscreen doesn't automatically start up in a few seconds, double-click autorun.exe in the rootdirectory of the CD.

Chapter 10

556

Select SQL Server 2000 Components.

Select Install Database Server.

Select Next.

Type in the SQL Server 2000 virtual server name.

Enter your name and company.

Read the software license agreement and, if you agree to the terms, select Yes.

Enter the SQL Server 2000 Enterprise Edition CD key.

Enter the IP address for the SQL Server 2000 virtual server you reserved prior to installation. Selectthe network connection (public or private); your SQL Server instance should use the publicconnection.

Select the shared disk that the data files should occupy. Do not use the quorum disk for your SQLServer data files; the Quorum disk belongs to the Cluster Group (or whatever you named the groupthat controls the failover cluster), which is the group that manages the cluster's groups and resources.If the SQL Server instance requires the cluster drive and the SQL Server group fails, this may also takethe cluster down. This is because dependencies, such as a physical disk, require that all dependents(including SQL Server and the Cluster Service) exist in the same group. Sharing the Quorum disk withyour Virtual SQL Server instance may cause I/O contention or performance issues.

Select which nodes in the cluster will have SQL Server installed. The default for a two-node clustershould already have the two physical nodes selected.

Select an administrator account with administrator permissions for both nodes in the cluster. Thismust be a domain account.

Decide whether or not this will be a default instance or a named instance. If a named instance,uncheck Default and select an instance name. For active/passive, keeping Default checked is the norm,but for active/active, your server names may make more sense if you use an instance name.

Select the installation type and directory location. Program files are placed on a local partition (duringinstallation on both nodes) and the data files on a shared disk.

Select which components to install.

Select the domain account to be used for the SQL Server and SQL Server agent services. This must bean account that exists on both nodes in the cluster, with administrator permissions.

Select the SQL Server instance collation.

Select the TCP/IP port. If the port number is 0, a port will be dynamically chosen for you uponinstallation. If this is an externally facing SQL Server Instance (accessed over a firewall), you maywish to pre-define a TCP/IP port that will be opened to the firewall for such SQL Server Instances. Ifyou are configuring a multi-instance cluster, you may need two such defined ports. Defining standardports for single or multi-instance clusters will allow you to minimize the number of ports that must beopened on your firewall for each externally-facing failover cluster. The alternative is unique ports,selected randomly during the install, for each instance belonging to a Virtual Server or NamedInstance.

Select Next:

Failover Clustering

557

Choose a licensing mode.

As the installation begins, you will receive update dialog boxes indicating that operations are beingperformed on the cluster nodes, and that virtual server resources are being created:

Select Finish.

Although you may not be prompted to do so, reboot the first node on which you installed the SQLServer 2000 Virtual Server, and then the second node too.

To create an active/active configuration, repeat the steps to add a new SQL Server instance.

10.2 How to… Install a Service Pack for a SQLServer 2000 Virtual Server

As new service packs are released, make sure to read the instructions thoroughly to see if any changes inprocedure have been added for clustered installs. The following example shows how to install service pack 2for SQL Server 2000 for a SQL Server 2000 Virtual Server (see Chapter 1 for a review of downloading andinstalling a service pack):

Chapter 10

558

First, make sure you are installing the service pack from the node currently controlling the SQL Serverinstance group that you wish to upgrade.

Double-click the setup.bat file in the installation path directory of the service pack files.

Select Next.

Select the virtual server name that you wish to upgrade.

Select the authentication mode for the service pack setup.

You will receive the following update:

Select the user name and password with administrator permissions for all nodes in the cluster.

Installation will begin. System databases will be updated, and binary files on both nodes will beupdated.

You will be prompted to back up the master and msdb databases.

Restart both nodes in the cluster. If you have an active/active topology, restart the first node, and thenfailover the non-upgraded instance to the restarted node and restart the second node. To fail over orfail back a group, simply right-click the group in Cluster Administrator and select Move Group. Failback each SQL Server group to its default instance (node by which you want each SQL Server group tobe controlled).

10.3 How to… Implement Post-InstallationSteps

After installing a SQL Server virtual server, there are some remaining steps that must be performed:

Start Cluster Administrator by selecting Start | Run and typing cluadmin.

In Cluster Administrator, type in the cluster or server name. If you have already been in ClusterAdministrator, any cluster connections you had open will immediately be brought up instead. If youwant to access a new cluster, go to the File | Open Connection window:

Give the SQL Server 2000 virtual server group a more appropriate name (instead of Disk Group 1 orsomething not indicating SQL Server). To change the group name, expand Groups and click the groupname once to enter a new name. Select Enter when complete.

Failover Clustering

559

Adding Additional Disk ResourcesAfter installing SQL Server, the only disk resource added to your SQL Server 2000 Virtual Server group willbe the data file drive selected during installation. If you wish to use additional shared disk drives for SQLServer, they must be added to the SQL Server group.

Open the group that currently holds the disk drives you wish to add to the SQL Server group,right-click each disk resource, select Change Group, and select the group to which to move it.

You will be prompted to make sure you wish to move this resource to a new group, and then again, forwhich you should select Yes:

If the original group is now empty, right-click the group in Cluster Administrator and select Delete.

Next, you must make this physical disk a dependency of SQL Server so that SQL Server can use it.Dependencies ensure that resources are available before a specific resource can be taken online withina cluster group. First, the SQL Server 2000 virtual server group must be taken offline; do this byright-clicking the group in Cluster Administrator, and selecting Take Offline.

In the right pane, with the Virtual SQL Server instance group selected in the left pane, right-click theSQL Server resource (the service) and select Properties.

In the Dependencies tab, click the Modify button.

Add the disk resource you wish the Virtual SQL Server instance to use, by clicking the availableresources and selecting the right arrow to add to the Dependencies pane.

Select OK in the Modify Dependencies dialog box, and OK again in the Properties dialog box for theservice.

Right-click the SQL Server group in the left pane of Cluster Administrator, and select Bring Online.

Other Post-Installation Configurations toMonitor

Watch out for fixed memory sizes defined on your SQL Server instances; make sure the total size usedfor an active/active cluster does not exceed the total memory resources of one node. For example, ifyou set a fixed 2.5GB size for each instance but the total memory per server is 4GB, you will haveproblems if two instances need to share a node.

Keep the BUILTIN\Administrators account in SQL Server, as it is used by the cluster serviceaccount.

If you plan on using replication on a clustered SQL Server instance, use a share name on the shareddisk. Do not use a local disk on either of the physical server nodes.

Always look for instructions specific to clustering. For example, if downloading the latest securitypatch, follow the instructions for a cluster and not the regular installation instructions. If explicitclustered installation instructions do not exist, do not assume that your changes will work, or thatyour new components will be cluster-aware.

If you ever need to change an IP address for the Virtual SQL Server, you must do so using the SQLServer Enterprise Edition installation CD. See Microsoft Knowledge Base article Q244980, HOWTO:Change the Network IP Addresses on a Virtual SQL Server.

To change the domain of a SQL Server Failover Cluster, reference the Microsoft Knowledge Basearticle 319016, HOW TO: Change Domains for a SQL Server 2000 Failover Cluster.

Chapter 10

560

Unlike a nonclustered instance of SQL Server 2000, renaming a Virtual Server on a failover cluster isnot recommended. You must uninstall and reinstall the cluster with the new name. For more details,see the Microsoft Knowledge Base article 307336, INF: How to Change a Clustered SQL Server NetworkName.

Microsoft recommends that symmetric multiprocessing (SMP) systems should have one processorreserved for the operating system and cluster service, using processor affinity. You should considerthis within the context of how intense CPU usage will be for your SQL Server 2000 virtual server (orservers for active/active).

10.4 How to… Troubleshoot a Failed SQLServer Virtual Server

When installing a new SQL Server 2000 virtual server, installation could take anywhere from 15 to 30minutes, depending on the hardware used. If your installation takes longer than that, make sure theinstallation has not hung. Press the Alt-Tab key combination to make sure that there are no error messages inthe background. For example, if attempting to install SQL Server from a network drive, you might receive afiles not found error in the background (which is why installing from a CD-ROM drive is preferred).

For failed or interrupted installations, you can usually restart installation by removing the resources createdso far via Cluster Administrator; these include the resources for SQL Server Name and SQL Server IP address.Do not delete the resource for the shared disk drive(s) you may be using. You may also need to delete binaryfiles that were installed on the local disk on both nodes, before trying to reinstall the SQL Server clusteredinstance.

Before attempting to do a manual cleanup of the files and resources, attempt a regular uninstall first (see thenext section). For more details on manual uninstalls of Virtual Servers, see the Microsoft Knowledge Basearticle 290991, Manually Remove SQL Server 2000 Default, Named, or Virtual Instance.

Some other troubleshooting tips to keep in mind:

SQL Server 2000 clustering does not support SQLMail and SQLAgentMail.

Examine the Sqlstp.log file in C:\WINNT directory for more detail about which errors occurred andhow far the installation proceeded before failing.

Check out the Microsoft Knowledge Base article Q321063 HOW TO: Troubleshoot the 'Setup Failed toPerform Required Operations on the Cluster Nodes' Error, for more in-depth information ontroubleshooting failed installations. Other useful troubleshooting articles include article Q279642,PRB: SQL Server 2000 Virtual Server Setup Error: 'The Drive Chosen for the Program Files InstallationPath <C:>, Is Not a Valid Path on All the Nodes of the Cluster', Not Valid; article Q235529, MSCSVirtual Server Limitations in a Windows 2000 Domain Environment; and article Q283794, ProblemsUsing Certificate with Virtual Name in Clustered SQL Servers.

10.5 How to… Uninstall a SQL Server 2000Virtual Server

To uninstall a SQL Server 2000 Virtual Server:

From the SQL Server 2000 Enterprise Edition installation CD, select Setup.

Failover Clustering

561

Select Next from the Welcome screen.

Select the SQL Server 2000 Virtual Server you wish to uninstall.

Select Upgrade, remove, or add components to an existing instance of SQL Server.

Select the named instance to remove.

Select Next to proceed with uninstalling the installation.

Select a valid service account with administrator permissions to all nodes (installation will removebinary files from both nodes in the cluster).

You will receive a prompt updating the progress of the operation.

You will then receive a prompt after the instance has been removed successfully.

Select Finish in the final dialog box. Reboot each node in the cluster.