7.6 Load Balance

www.bmc.com

White Paper

Using a Hardware Load Balancer with AR System 7.6.04

January 2011

If you have comments or suggestions about this documentation, contact Information Development by email at [email protected].

Contacting BMC Software

You can access the BMC Software website at http://www.bmc.com. From this website, you can obtain information about the company, its products, corporate offices, special events, and career opportunities.

United States and Canada

Address BMC SOFTWARE INC2101 CITYWEST BLVDHOUSTON TX 77042-2827 USA

Telephone 713 918 8800 or800 841 2031

Fax 713 918 8000

Outside United States and Canada

Telephone (01) 713 918 8800 Fax (01) 713 918 8000

© Copyright 2004–2011 BMC Software, Inc.

BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners.

IBM, Informix, DB2, and AIX are registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Linux is the registered trademark of Linus Torvalds.

UNIX is the registered trademark of The Open Group in the US and other countries.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

BMC Software considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable End User License Agreement for the product and the proprietary and restricted rights notices included in this documentation.

Restricted Rights Legend

U.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC Software, Inc., 2101 CityWest Blvd., Houston, TX 77042-2827, USA. Any contract notices should be sent to this address.

http://www.bmc.com

mailto:[email protected]

Customer Support

You can obtain technical support by using the Support page on the BMC Software website or by contacting Customer Support by telephone or email. To expedite your inquiry, please see “Before Contacting BMC Software.”

Support Website

You can obtain technical support from BMC Software 24 hours a day, 7 days a week at http://www.bmc.com/support_home. From this website, you can

■ Read overviews about support services and programs that BMC Software offers.■ Find the most current information about BMC Software products.■ Search a database for problems similar to yours and possible solutions.■ Order or download product documentation.■ Report a problem or ask a question.■ Subscribe to receive email notices when new product versions are released.■ Find worldwide BMC Software support center locations and contact information, including email addresses, fax

numbers, and telephone numbers.

Support by telephone or email

In the United States and Canada, if you need technical support and do not have access to the Web, call 800 537 1813 or send an email message to [email protected]. (In the Subject line, enter SupID:<yourSupportContractID>, such as SupID:12345.) Outside the United States and Canada, contact your local support center for assistance.

Before Contacting BMC Software

Have the following information available so that Customer Support can begin working on your issue immediately:

■ Product information

— Product name— Product version (release number)— License number and password (trial or permanent)

■ Operating system and environment information

— Machine type— Operating system type, version, and service pack— System hardware configuration— Serial numbers— Related software (database, application, and communication) including type, version, and service pack or

maintenance level

■ Sequence of events leading to the problem

■ Commands and options that you used

■ Messages received (and the time and date that you received them)

— Product error messages— Messages from the operating system, such as file system full— Messages from related software

http://www.bmc.com/support_home

mailto:[email protected]

White Paper


This white paper describes how to use a hardware load balancer to improve the scalability and availability of the BMC Remedy Action Request System (AR System).

The following topics are provided:

Overview (page 6)Functionality (page 7)Considerations for configuration with AR System servers (page 8)Configuration examples (page 8)Special considerations and known issues (page 13)Sample load-balancer: Cisco 11500 Series Content Services Switch (page 20)

Using a Hardware Load Balancer with AR System 7.6.04 5

White Paper

OverviewHigh-load environments present special issues relating to performance, hardware capacity, system downtime, and so on. A high-load environment presents two major issues:

System scalability is dependent upon the capacity of the hardware. If the existing hardware is at its limit, it needs to be replaced by bigger, more powerful hardware. New hardware is often much more expensive than existing hardware. The old hardware could only be used as a “hot” standby system. (A hot standby system is running and ready for immediate service. It can be switched into service manually or automatically upon detection of a failure in the active equipment.)

The system needs to be available for as much time as possible. This can be challenging and often requires redundant systems. The backup system is often in hot standby mode and is only activated in the event of an outage. Only one system can be used in production.

SolutionThe solution to both challenges is a technology commonly used in the web environment—a load balancer. You can use a hardware load balancer to improve the scalability and availability of AR System servers.

A load balancer is a valuable component in building a scalable, highly available AR System infrastructure. Scalability is achieved through the ability to add AR System servers as demand and workload increase. High availability is achieved by configuring multiple AR System servers to handle client requests, and if one AR System server fails, other AR System servers in the group handle the additional workload. By creating an infrastructure that scales with workload and reduces downtime, you can maximize the return on your AR System investment.

DefinitionA load balancer is a transparent network device that distributes the traffic from multiple clients to several servers, preventing AR System servers from becoming overloaded. Distributing workload among several AR System servers can lead to performance benefits and cost savings. Buying many smaller machines is far less expensive than paying for a single high-performance machine. Since load balancers redirect incoming traffic, they are sometimes referred to as “redirectors.”

A load balancer also provides high availability through redundancy and fail-over protection. Fail-over is the process of moving operations from one server to another upon service or machine failure. If one AR System server becomes unavailable for software reasons or if the machine hardware fails, other AR System servers in the group (or cluster) can take over the workload. Users will not be aware of any problems, nor will they experience any downtime.

6 Using a Hardware Load Balancer with AR System 7.6.04


Most load balancers work on the TCP level of the network protocol and can therefore balance the load of many applications that use this protocol. Some examples include HTTP, FTP, and the AR System server. The load balancer is transparent to users, so the client application cannot “see” it and is unaware that the client requests are being load-balanced.

You can think of the load balancer as a virtual system, or as a “super” AR System server in this case. The load balancer is given a virtual IP address and a DNS name, either of which is used by BMC Remedy User clients when connecting to the group of load-balanced AR System servers. Both the short and long DNS names must be resolvable. (Long DNS names are fully qualified with a domain.) When a client request is received, the load balancer passes the request to one of the actual AR System servers within the group. The chosen AR System server performs the work and sends the results to the client. This balancing of connections spreads out the client requests evenly and distributes the load.

NOTE For performance reasons, the AR System API clients establish a connection with the AR System server and keep the connection until the session is terminated or the connection is interrupted.

FunctionalityLoad balancers forward data packets in the same manner as Network Address Translation (NAT) devices. The packets are forwarded at the TCP level with modified address information to the target system.

To distribute client requests across each group of AR System servers, the load balancer can use various scheduling policies. The following two policies are the most common:

Round Robin—This policy directs client requests to each AR System server, one at a time. Successive requests are routed to the next AR System server, then the next, and this process is repeated consecutively.

Least Connections—This policy directs client requests to the AR System server that has the fewest connections.

For better throughput, most load balancers offer multiple network ports. This method avoids collisions between the incoming and outgoing routed network traffic.

The load balancer also offers clients the ability to stick to one target system. This means that all requests from a single client are routed to the same system.

Functionality 7

White Paper

For versions 7.6.04 or later, BMC recommends configuring the load balancer that is located between the web servers and AR System servers without setting a “sticky” bit. In versions earlier than 7.6.04, BMC recommended setting the sticky bit to activate session affinity to route all connections from one web server to the same AR System server. For more information, see “Scenarios for load balancing between the web servers and AR System servers” on page 14 and the BMC Remedy Mid Tier Guide.

Considerations for configuration with AR System servers

The load balancer acts like a NAT device that routes any TCP or UDP traffic. Since the AR System server uses an ONC-RPC implementation that is layered on top of TCP/IP, AR System server traffic can be load balanced. Because of the nature of the client/server interaction within AR System, setting the sticky bit is not required. While the sticky bit allows for the spreading of the workload from multiple clients, it does reduce the balancing that can occur.

AR System includes the capability for automatic fail-over of special operations and the sharing of floating licenses among the servers. Server groups are independent of load balancing, but the concepts are complementary.

To enable load balancing across multiple AR System servers, configure each server as outlined in the Configuration Guide.

Configuration examplesThe following section describes a number of common configuration examples. All configurations require that AR System servers have been properly configured. Verify that the following configuration steps have been completed before you review the examples:

1 All AR System servers share the same database.

2 All AR System servers have the same server name (server name alias), and they have a unique name configured for server-to-server communication.

3 All AR System plug-ins are configured to access the local server.

4 All AR System servers are configured to listen on TCP ports that the load balancer is configured for.

5 External operations managed by the server group (such as Email Engine and Flashboards) must be must be installed locally on the AR System servers that manage them.



6 BMC Remedy Alert is configured for a load-balanced environment.

The load-balancer IP address is mapped to the AR System server IP address.

The server is configured to ignore the actual IP address recorded during Alert client registration.

7 Distributed Server Option (DSO) is configured for a load-balanced environment.

Mappings have been updated to take advantage of the server group.

8 Full text search (FTS) is configured for a load-balanced environment.

The index collection directory is accessible to all AR System servers in the group.

9 Clients are configured to use the virtual address and port of the load balancer.

10 Servers are declared to be server group members, and operation rankings are configured.

Configuring a load balancer with multiple AR System serversIn this example, the load balancer is directing traffic to three AR System servers that share a single database. In Figure 1-1, the uppermost AR System server has the primary ranking for the Administration and Escalation operations. The other two AR System servers can be used to back up these operations, when the uppermost server is not running.

Figure 1-1: Basic load-balancer configuration with multiple AR System servers

Configuring a load balancer with a firewall and multiple AR System servers

In this example, client requests pass through a firewall and into the load balancer. The load balancer directs traffic to three AR System servers. The three AR System servers share a single database.

In Figure 1-2, the uppermost AR System server has the primary ranking for the administration operation, but the bottommost AR System server has the primary ranking for the escalation operation. As mentioned earlier, the administrative AR System server can be a computer other than the escalation AR System server.

Configuration examples 9

White Paper

Figure 1-2: Load-balancer configuration with a firewall and AR System servers

In Figure 1-3, AR System server client traffic comes into the firewall on TCP port 3030 and is directed to the load balancer. The load balancer routes this traffic to one of the AR System servers listening on port 2020. As shown in the diagram, the port number for the virtual address can be different from the port numbers for the real AR System servers.

Figure 1-3: Load-balancer configuration with a firewall, a virtual AR System server, and real AR System servers

Configuring a load balancer with a firewall, web servers, and multiple AR System servers

In this example, client requests pass through a firewall and into the load balancer. The load balancer directs the web requests to three web servers. The web servers access the AR System servers to fulfill AR System requests. The three AR System servers share a single database, as shown in Figure 1-4.



Figure 1-4: Load-balancer configuration with a firewall, web servers, and AR System servers

Using the hosts file, the IP address of the AR System server needs to be resolved manually on each web server. This is necessary because all web servers reference the same AR System server name; however, each web server is linked to a different AR System server, and each AR System server has a different IP address. If the server name was resolved using a DNS server, this server name would resolve to the same IP address and all the web servers would communicate with the same AR System server. Therefore, each web server uses the hosts file, and manually resolves the AR System server name to the appropriate AR System server.

On Windows platforms, the hosts file is located in the %WINDIR%\system32\drivers\etc directory. On UNIX, the hosts file is located in the /etc directory.

The hosts file on Web 1 should include the following line: myarserver 11.40.35.47

The hosts file for Web 2 should include the following line: myarserver 11.40.35.49

The hosts file for Web 3 should include the following line: myarserver 11.40.35.51

In the preceding sample hosts files, myarserver is the AR System server name that all the web servers reference.

Configuring a load balancer with a firewall, web servers, a second load balancer, and multiple AR System servers

In this example, client requests pass through a firewall and into a load balancer. The first load balancer directs web requests to the web servers. Web server requests to the AR System servers are directed through a second load balancer. The three AR System servers share a single database, as shown in Figure 1-5.

Configuration examples 11

White Paper

Figure 1-5: Load-balancer configuration with a firewall, web servers, AR System servers, and two load balancers

NOTE For versions 7.6.04 or later, BMC recommends configuring the load balancer that is located between the web servers and AR System servers without setting a sticky bit. In versions earlier than 7.6.04, BMC recommended setting the sticky bit to activate session affinity to route all connections from one web server to the same AR System server. For more information, see “Scenarios for load balancing between the web servers and AR System servers” on page 14 and the BMC Remedy Mid Tier Guide.

This type of configuration can also be used with a WAN, DMZ, or LAN as shown in Figure 1-6. Client requests pass through the firewall and into the first load balancer. The load balancer routes the traffic to one of the web servers. AR System requests from the web servers pass through the first load balancer, then through the firewall, and finally to the second load balancer. The second load balancer then routes the request to one of the AR System servers in the group.

Figure 1-6: Load-balancer configuration with a WAN, a firewall, web servers, AR System servers, and two load balancers



For better throughput, such as might be required in a high-performance environment, you can add a second firewall, as shown in Figure 1-7. AR System server traffic from the web servers is routed through the second firewall. AR System server traffic from web servers follows a different route from that of incoming AR System client requests, thereby reducing the likelihood of a network bottleneck in the load balancer.

Figure 1-7: Load-balancer configuration with a WAN, two firewalls, web servers, AR System servers, and two load balancers

Special considerations and known issuesThe following sections provide information about situations you might encounter, as well as descriptions of issues that have been identified with respect to using a load balancer with AR System.

Fail-over of escalation and archive operationsIf an Escalation or Archive operation results in a fail-over from one server to another, Escalations or Archives running at a fixed time might be skipped or might run twice. An Escalation or Archive activity might be skipped if it is scheduled to run while the operation is making the transition from one server to another. An Escalation or Archive activity might be skipped or run twice if the system clocks on the relevant systems are skewed—that is, if the clock settings differ enough to cause a fixed time to be missed or encountered twice.

Special considerations and known issues 13

White Paper

Scenarios for load balancing between the web servers and AR System servers

For versions 7.6.04 or later, BMC recommends configuring the load balancer between the web servers and AR System servers without setting a sticky bit. This allows a session from any web server to be distributed to any AR System server.

This section describes the configuration for a load balancer between the web servers and AR System servers without setting a sticky bit for version 7.6.04 or later. As a starting point, the configuration for setting a sticky bit for the load balancer for versions earlier than 7.6.04 is first discussed.

Load balancing between the web servers and AR System servers by setting a sticky bitIn versions earlier than 7.6.04, BMC recommended configuring the load balancer between the web servers and AR System servers by setting a sticky bit. In this configuration, the load balancer between the web servers and the AR System servers routed all connections from one web server to the same AR System server, resulting in session affinity.

Figure 1-8 illustrates session affinity between the web servers (WS) and the AR System servers (ARS) when the load balancer (LB) is configured with a sticky bit. For example, session affinity is established between WS1 and ARS1, WS 2 and ARS2, and WS3 and ARS3. The load balancer routes all connections from WS1 to ARS1.



Figure 1-8: Load-balancer configured with the sticky bit between the web servers and AR System servers

Adding an AR System server without setting a sticky bitFigure 1-9 shows an example AR System with three web servers and two AR System servers where session affinity has been established between WS1 and ARS1, WS2 and ARS2, and WS3 and ARS1.

Figure 1-9: Load-balancer configuration with three web servers and two AR System servers with session affinity established

If a new AR System server (ARS3) is added to this AR System, ARS3 is never considered as available because WS1, WS2 and WS3 have already established session affinity to either ARS1 or ARS2 (Figure 1-10).

ARS1

ARS2

ARS3

WS1

WS2

WS3

LB with sticky bit

LEGENDARS: AR System serverLB: Load balancerWS: Web server

ARS1

ARS2

WS1

WS2

WS3

LB with sticky bit


White Paper

Figure 1-10: Load-balancer configuration with a new AR System server added to three web servers and two AR System servers with session affinity established

In version 7.6.04 or later, you can route connections from a web server to a new AR System server on the fly by selecting the Enable Lifespan setting on the Connection Settings page of the Mid Tier Configuration Tool (Figure 1-11). This setting enables load balancing without setting a sticky bit. For more information, see the BMC Remedy Mid Tier Guide.

Figure 1-11: Enable Lifespan setting on the Connection Settings page of the Mid Tier Configuration Tool

With Enable Lifespan selected, the load balancer distributes sessions from any web server (in this case, WS3) across the AR System server group according to its scheduling policy (round robin or least connections). The newly-available AR System server (ARS3) is considered within that distribution, as shown in Figure 1-12. Connections may or may not get routed to ARS3 depending on the scheduling policy.

ARS1

ARS2

WS1

WS2

WS3 ARS3 new

LB with sticky bit



Figure 1-12: Load-balancer configuration with a new AR System server added without setting a sticky bit for the load balancer

Reestablishing an AR System server after fail-over without setting a sticky bitIn AR System versions earlier than 7.6.04 (Figure 1-8), suppose that ARS3 fails. If a sticky bit is configured for the load balancer, the following occurs:

The load balancer no longer uses ARS3 as an active resource.

The ARS3 end user receives an error.

The connection from WS3, which was previously routed to ARS3, is routed to ARS1 or ARS2. In this example, WS3 sessions go to ARS1.

Figure 1-13 illustrates the resulting unbalanced connections between the web servers, load balancer, and AR System servers.

Figure 1-13: Load-balancer configuration with web servers and AR System servers with the sticky bit set for the load balancer when an AR System server fails

When ARS3 recovers and is recognized as an active resource by the load balancer, it does not receive connections from any of the three web servers because session affinity has been established between the web servers and either ARS1 or ARS2 (Figure 1-10). To rebalance the servers in versions earlier than 7.6.04, you need to shut down the AR System servers, load balancer, and web servers and then restart them in the proper sequence.

ARS1

ARS2

ARS3 new

WS1

WS2

WS3

LB without sticky bit

ARS1

ARS2

WS1

WS2

WS3 ARS3 fails

LB with sticky bit


White Paper

In version 7.6.04 or later, make sure the Enable Lifespan check box is selected on the Connection Settings page of the Mid Tier Configuration Tool (Figure 1-11). If a sticky bit is not set, the load balancer distributes sessions from any web server to any AR System server, including a recovered or new AR System server (Figure 1-12).

Limiting idle session connectionsIn AR System versions earlier than 7.6.04 (Figure 1-8), the idle connections between a web server and AR System server can stay open indefinitely. Performance problems from this can occur under certain circumstances. For example:

An AR System server is removed from the server group.

The maximum number of sessions for an AR System server is reached during a resource spike.

AR System servers users leave their sessions open for extended period, such as during a lunch break or at the end of the day.

The source from these open idle sessions connections is not redistributed.

Figure 1-14: Load-balancer configuration with web servers and AR System servers with the sticky bit set for the load balancer when an AR System server fails

To avoid problems from idle connections in version 7.6.04 or later, you can configure the fields in the Connection Pool Settings section on the Connection Settings page in the Mid Tier Configuration Tool (Figure 1-15). The Idle Connections per Server setting allows you to limit the number of idle connections per server. The Connection Timeout field allows you to specify how long the connections exceeding that limit will remain open before being terminated. For more information, see the BMC Remedy Mid Tier Guide.

Lowering the Idle Connections per Server value minimizes the number of idle connections that can stay open until their sessions ends. If the Idle Connections per Server field is set to 0, then the connection pool will be closed when all connections are closed.

ARS1

ARS2

WS1

WS2

WS3 ARS3 fails

LB with sticky bit

Idle sessions fromany web server thatwere routed to AR3

will remain open



Lowering the Connection Timeout value minimizes the time that idle session-based connections can stay connected before being closed, resulting in better load redistribution. The number of current idle connections is determined by whether the Connection Time or Connection Lifespan setting is first met.

Figure 1-15: Connection Pool Settings on the Connection Settings page of the Mid Tier Configuration Tool

Configuring the Connection Pool Settings allows idle sessions to be used again in a timely manner. The AR System server end users see no change in behavior because their connections are not dropped.

Load balancing between the clients and web servers without setting a sticky bit

If the web servers in your AR System were installed in a cluster (with fail-over enabled), then setting the sticky bit on the load balancer between the clients and web servers is not needed.

Server configuration sharingThere is no provision in AR System for the sharing of common configuration information among servers in a multiple-server environment. Therefore, you must synchronize common configuration items. For example, changes made to the configuration file of one server must be propagated to the other servers.


White Paper

Licensing servers in a multiple-server environmentIn an environment where several AR System servers share a single database, each server must be licensed separately. You must purchase server licenses for your primary server and each additional server in your environment. However, other licenses on the primary server (such as floating and DSO licenses) are extended to additional servers at no charge.

Sample load-balancer: Cisco 11500 Series Content Services Switch

The BMC Software Quality Assurance department tested a load-balancing configuration using the Cisco 11500 Series Content Services Switch (CSS) with software version WebNS 4.10, 5, 6, 7, build 17s. The two AR System servers shared a database on another computer, as shown in Figure 1-16. Specific ports were not defined, and the default setting (ON) was used for the persistence (sticky) option. The default setting of round robin was used for our load-balancing technique.

Figure 1-16: Load-balancer configuration tested by BMC Software

The following configuration was used for the CSS appliance:

!*************************** GLOBAL ***************************username admin1 des-password <encrypted_password> superuserusername admin2 des-password <encrypted_password> superuserusername admin3 des-password <encrypted_password> superuser

ip route 0.0.0.0 0.0.0.0 172.23.32.1 1

!************************* INTERFACE *************************interface e5

bridge vlan 10

interface e6bridge vlan 20

!************************** CIRCUIT **************************



circuit VLAN10

ip address 172.23.32.15 255.255.248.0

circuit VLAN20

ip address 172.23.41.5 255.255.255.0

!************************** SERVICE **************************service www-server1

ip address 172.23.41.15active

service www-server2ip address 172.23.41.16active

!*************************** OWNER ***************************owner sample

content rule1add service www-server1add service www-server2vip address 172.23.33.95active

For information about how to configure a load balancer with the Cisco CSS, see the Cisco Systems website at http://www.cisco.com. The following documents are relevant to load-balancer configuration:

“Basic CSS Load Balancing Configuration,” Document ID 12557, http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_configuration_example09186a008009438d.shtml

“CSS Basic Configuration Guide,” Text Part Number 78-13886-05, http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11500series/v7.20/configuration/basic/guide/basicgd.pdf

NOTE Website addresses change frequently. If you have difficulty finding these documents, go to http://www.cisco.com and navigate to the Documentation pages.

Sample load-balancer: Cisco 11500 Series Content Services Switch 21

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11500series/v7.20/configuration/basic/guide/bsccfggd.html

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11500series/v7.20/configuration/basic/guide/bsccfggd.html

http://www.cisco.com

http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_configuration_example09186a008009438d.shtml

http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_configuration_example09186a008009438d.shtml

http://www.cisco.com

White Paper


*186198**186198**186198**186198*

*186198*

Documents

7.6 Load Balance