Troubleshoot Cisco Networks Ch15

8/8/2019 Troubleshoot Cisco Networks Ch15

http://slidepdf.com/reader/full/troubleshoot-cisco-networks-ch15 1/46

617

ProLib / Cisco: A Beginner’s Guide. 4/E/Velte & Velte/6383-0/Chapter 15

15

Troubleshooting

Cisco Networks

VelteCh15V3.indd 617VelteCh15V3.indd 617 11/4/06 8:44:12 PM11/4/06 8:44:12 PM




618 Cisco: A Beginner’s Guide, Fourth Edit ion

Keeping an internetwork going is a full-time job. As you saw earlier, problemsemerge with such frequency that the industry invented routing protocols todeal with them automatically, not even waiting for a network administrator to

intervene. They do a pretty good job, at least with problems that can be ameliorated bydetouring to a new route. But changing routes is only a temporary solution. In order fora network to run effectively, all its components must be running properly and constantly.After all, delivering available bandwidth to users under normal circumstances is hardenough without having a LAN segment out of commission or a router functioning at

partial capacity.For this reason, a big part of a network team’s time is spent troubleshooting.Problems range from a single user unable to access a service to an entire LAN segmentcrashing. Troubleshooting isn’t just a matter of finding and fixing broken parts; much ofit is dedicated to fixing performance bottlenecks. When a problem emerges, the networkadministrator often has no idea which device is causing the trouble. And once the problemdevice is identified, the cause of the problem must be diagnosed. Then decisions must bemade on how to fix the situation.

A methodical approach should be taken to troubleshooting; otherwise, a lot of time can

be wasted trying to figure out what’s causing the problem. Like a doctor, the troubleshootermust recognize the symptoms, associate them with a set of probable causes, and thenprogressively narrow down the list until the culprit is finally identified. From there, aproper action plan must be devised and implemented. That’s troubleshooting.

In this chapter, we’ll review how to troubleshoot problems in a variety of Ciscoconfigurations by running through some troubleshooting scenarios. For simplicity’ssake, we’ll assume IP as the network protocol and the Microsoft Windows platformas the host. Although the terminology can vary, networking problems are largely thesame, regardless of the protocol or host environment. We’ll also restrict the examples totroubleshooting routers, which is where most of the action is.

THE MECHANICS OF NETWORK TROUBLESHOOTING In internetworks, trouble is often caused either by failing device hardware or a configurationproblem. The location of most problems can be identified remotely, and to some extent,the problems can also be diagnosed and even fixed remotely (but the hardware must still be running for that). By “fixing remotely,” we mean without walking over and actuallyinspecting and touching the device; we don’t necessarily mean being geographicallyremoved. If, say, an enterprise’s campus internetwork is experiencing a problem, networkadministrators usually do most troubleshooting tasks without even leaving their desks.

In Cisco environments, remote work can be done through a network managementconsole or by logging directly into a device’s IOS command-line environment throughTelnet or Secure Shell (SSH). As you learned, the Cisco NMS consoles—CiscoWorksResource Manager Essentials and CiscoWorks Campus Manager—use their graphical in-terfaces to indirectly manipulate IOS commands inside the remote device. Thus, most ofthe real troubleshooting work takes place inside the device’s IOS environment. Here arethe major IOS commands used to perform most troubleshooting tasks:





619Chapter 15: Troubleshooting Cisco Networks

▼ ping Indicates whether “echo” packets are reaching a destination andreturning. For example, if you enter ping 10.1.1.1, IOS will return the percent-age of packets that echoed back from the 10.1.1.1 interface.

■ traceroute Reports the actual path taken to a destination. For example, if youenter traceroute ip 10.1.1.1, IOS will list every hop the message takes to reachthe destination 10.1.1.1 interface.

▲ show Reports configuration and status information on devices and networks.For example, the show memory command displays how much memory isassigned to each network address and how much is free.

The source of problems must be in either device or network media (cabling, connec-tors, and so on). Even if the trouble is in a cable, the way to it is through IOS. The ping and trace commands are used to locate problems. If the device is still running, the show and debug commands are employed to diagnose them. Actual fixes are done by chang-ing either the hardware or its configuration. The debug command is similar to show,except it generates far more detailed information on device operations—so much so thatrunning debug may greatly slow down device performance.

Network Troubleshooting Methods Problems are usually brought to a network administrator’s attention by users. Theywant to know why they can’t access a service within the enterprise’s internetwork, orthey complain that performance is slow. The location and nature of the complaint arethemselves strong clues as to what’s causing the problem. Many times, the administra-tor immediately knows what’s wrong and how to fix it, but oftentimes an investigationmust be launched to figure out which device is the source of the trouble, what’s causing

it, and what the best way to fix it is. The network administrator must find answers bymethodical troubleshooting. As you might imagine, troubleshooting largely works by aprocess of elimination, as in the following:

▼ What are the symptoms? Usually, this boils down to users not being ableto reach a destination. Knowing both endpoints of a network problem—thesource and destination addresses—is the base information in most trouble-shooting situations.

■ Where do I start looking? Does the scenario fit a known pattern that suggestsprobable causes? For example, if a server isn’t responding to service requestsfrom a client, there could be a problem with the server or the client itself. If theserver is working okay for other clients, then it might be the client device. If notthat, then the problem must reside somewhere between the two.

■ Where do I start? There are rules of thumb that short-list what’s most likelycausing a certain type of symptom. The administrator should diagnose “best-candidate” causes first. For example, if a server accessed over a WAN linkseems slow to remote dial-in users, the link could be going bad, usage could be






up, there could be a shortage of buffer memory in the router interface servicingthe link, or the hosts could be misconfigured. One of these probable causes willexplain the problem 95 percent of the time.

▲ What’s the action plan? Finding the exact cause of a problem in amalfunctioning device means dealing with one variable at a time. For example,it wouldn’t make sense to replace all network interface modules in a router before rebooting. Doing so might fix the problem, but it wouldn’t definethe exact source or even what fixed it. In science, this is called changing one

variable at a time. The best practice is to zero in on the source by cuttingvariables down one by one. That way, the problem can be replicated, the fixvalidated as a good one, and the exact cause recorded for future reference. Anaction plan also allows you to undo changes that don’t fix the problem (or mayeven make it worse).

NOTE Before you get too bogged down in trying to isolate the problem with checking IP addresses or other configuration information, check the cables. You might save yourself hours of trouble and

effort by reconnecting a loose cable or a power cord that has come undone.

Most internetwork problems manifest themselves as either seriously degraded per-formance or as “destination unreachable” timeout messages. Sometimes, the problem iswidespread; other times, it’s limited to a LAN segment or even to a specific host. Let’stake a look at some typical problems mapped to their probable causes. Table 15-1 outlinesproblems with host connectivity. (Hosts are usually single-user PCs, but not always.)

Unfortunately, most internetwork problems aren’t limited to a single host. If a prob-lem exists in a router or is spread throughout an area, many users and servers are affected.Table 15-2 outlines a couple of typical network problems that are more widespread.

Table 15-1. Typical Host Access Problems and Causes

Symptoms Probable Causes

Host can’t accessnetworks beyond localLAN segment.

Misconfigured settings in host device, such as bad default gateway IP address or bad subnetmask.

The gateway router is malfunctioning.

Host can’t access

certain services beyondlocal LAN segment.

Misconfigured extended access list on a router

between the host and the server. Misconfiguredfirewall, if the server is beyond the autonomoussystem.

The application itself may be down.






Many times, networks and services are reachable, but performance is unacceptablyslow. Table 15-3 outlines factors that can affect performance within a local network. Itdoesn’t address WAN links, however. They’re covered separately later in this chapter because serial lines involve a slightly different set of technologies and problems.


Most users can’taccess a server.

Misconfigured default gateway specificationin the remote server.

Misconfigured access list in the remote server.

Hosts unable to obtain IP addresses throughDHCP.

Connections to an areacan’t be made when onepath is down.

Routing protocol not converging within therouting domain.

All interfaces on router handling alternativepath not configured with secondary IPaddresses (discontinuous addressing).

Static routes incorrectly configured.

Table 15-2. Typical Router Problems and Causes


Poor server response;hard to make andkeep connections.

Bad network link, usually caused by amalfunctioning network interface module orLAN segment medium.

Mismatched access lists (in meshedinternetwork with multiple paths). Congestedlink, overwhelmed by too much traffic.

Poorly configured load balancing (routingprotocol metrics).

Misconfigured speed or duplex settings.

Table 15-3. Campus LAN Performance Problems and Causes






Troubleshooting Host IP Configuration If a user is having trouble accessing services and the overall network seems to be okay, agood place to start looking for the cause of the problem is inside that person’s computer.There are a couple of things that could be misconfigured in the user ’s host computer:

▼ Incorrect IP information The IP address or subnet mask information could be missing or incorrect.

■ Incorrect default gateway The default gateway router could be

misconfigured.

▲ Nonfunctioning name resolution DNS or WINS could be misconfigured.

To refresh on the subject, every host has a default gateway specified in the host’s net-work settings. A default gateway is an interface on a local router that is used for passingmessages sent by the host to addresses beyond the LAN. A default gateway (also calleda gateway of last resort) is configured, because it makes sense for one router to handlemost of a host’s outbound traffic in order to keep an updated cache on destination IP ad-dresses and routes to them. A host must have at least one gateway, and a second one is

sometimes configured for redundancy in case the primary gateway goes down.

Checking the Host IP Address Information Misconfigured network parameters in desktop hosts are usually attributable to a mistake by the end user. Keep in mind that—on Windows computers, at least—administratorsand power users can easily access and modify network settings. To check the host’s IPaddress information in Windows XP, for example, click the Start button on the menu bar,choose Run, type cmd, and then click OK. This will open the Windows XP commandprompt window. At the command prompt type ipconfig /all and then press, <enter>.

This will display the host’s IP address configuration.If you need to configure the host’s IP address settings, this is done through the Net-

work Properties screen. To access the network properties in Windows XP, for example,click the Start button on the menu bar, and then choose Connect To | Show All Connec-tions. From the resulting list of network connections, right-click the appropriate networkicon, and then select Properties. Click the General tab, and in the This Connection Usesthe Following Items box, click Internet Protocol (TCP/IP), and then click the Properties button. In Windows NT/2000, click the Start button on the menu bar, and then chooseControl Panel | Network | Configuration, and, finally, TCP/IP Properties. This will al-

low you to set the IP address and default gateway.The protocol will usually point to a network interface card (NIC) connecting the host

to the LAN, as is the case with the TCP/IP Ethernet PC card highlighted in Figure 15-1.(If the host dials into the internetwork, the protocol that points to the dial-up adaptershould be selected instead.)

Once you’re pointed at the right NIC, start by making sure that the host is identifyingitself correctly to the network. The example in Figure 15-2 shows a statically defined IPaddress and subnet. These must match what’s on file for the host in the config file of the






router serving as the default gateway. If the Obtain An IP Address Automatically check box is selected, the host’s IP address is dynamically assigned by a server—a DynamicHost Control Protocol (DHCP) server. While DHCP can make address assignment mucheasier and somewhat foolproof, rogue DHCP servers can be problematic, so be sure tocheck IP address settings to verify which DHCP server is assigning the IP address usingthe ipconfig/all command.

Next, make sure the host’s declared IP address is the right one by logging into its

gateway router and entering the show arp command. You’ll remember that ARP standsfor Address Resolution Protocol, a utility that maps the physical device’s media accesscontrol (layer 2) address to its assigned IP (layer 3) address in order to handle the finalstage of delivery between the gateway router and the host. Figure 15-3 shows the ARPtable in the config file of our example gateway router. The shaded line shows that theEthernet interface indeed has an address 10.1.13.12 on file, as was declared in the host’sIP Address tab. The MAC address can also be verified using the ipconfig/all command.

Figure 15-1. To troubleshoot a host, the place to start is the network interface card






Another potential host problem is the config settings for the default gateway itself. Inother words, you have to make sure the host has the correct IP address configured as itsdefault gateway router, as shown in Figure 15-4.

The host’s default gateway IP address must match the one set for the networkinterface module on the gateway router. To check that this is the case, go to the gatewayrouter and enter the show interfaces command, as shown here:

MyRouter#show interfaces

.

.

.

Ethernet1 is up, line protocol is up

internet address is 10.1.13.1/28

ip accounting output-packets

ip nat inside

ip ospf priority 255

Figure 15-2. The host’s IP address settings and those in the default gateway must match






media-type 10BaseT

.

.

.

As you can see, our example interface, Ethernet1, is indeed addressed 10.1.13.1, asdeclared in the host’s Gateway tab.

This is also where you can check to make sure the host’s declared subnet mask

matches the one on file in the gateway router. Mask/28 is also correct, because it matchesthe one (255.255.255.240) declared in the host’s IP Address tab.

Obviously, if any of the host’s network settings are incorrect, the administrator shouldadjust them to match the gateway router’s settings, reboot the PC, and try to make anetwork connection. On the other hand, if the PC’s settings are okay, the troubleshootermust work outward from the operable host to identify the source of the problem.

Isolating Connectivity Problems

Most network problems have to do with the inability to connect to a desired host orservice. Connectivity problems—also called “reachability problems”—come in manyforms, such as attempted HTTP connections timing out, attempted terminal connectionsgetting no response from the host, and so on. As just outlined, the troubleshooter shouldfirst make sure the host reporting the problem is itself properly configured, and thenwork outward. To draw an analogy, the troubleshooter must work the neighborhooddoor to door, much like a cop searching for clues.

Figure 15-3. The host’s IP address must match the one for the gateway router in the ARP file

vsigate#show arp

Protocol address Age (min) Hardware Addr Type Interface

Internet 10.1.13.11 0 0050.0465.395c ARPA Ethernet1

Internet 10.1.11.1 12 0060.3eba.a6a0 SNAP TokenRing0

Internet 10.1.13.12 9 00a0.c92a.4823 ARPA Ethernet1

Internet 10.1.11.2 - 0006.f4c5.5f1d SNAP TokenRing0

Internet 10.1.11.3 190 0006.c1de.4ab9 SNAP TokenRing0

Internet 10.1.12.3 - 0006.f4c5.5fdd SNAP TokenRing1

Internet 10.1.13.12 15 0050.04d7.1fa4 ARPA Ethernet1

MAC address






Checking Between the Host and Its Gateway Router If the host’s network settings are configured properly, the next step is to work outwardfrom the host to the gateway router. This should be done even if the host’s problem isfailing to connect to a remote server. Before working far afield, the best practice is to firstcheck the link between the host and its gateway router.

Using the ping Command The easiest way to check a link is to use the ping command.This command sends ping packets to a specific network device to see if it’s reachable. Intechnical terms, ping sends its packets through the ICMP transport protocol instead of

through UDP or TCP. It actually sends several packets, as shown here:MyRouter#ping 10.1.1.100

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.1.1.100, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

MyRouter#

Figure 15-4. Check to make sure the correct default gateway IP address is configured






Host computers and network devices both have ping commands. The precedingexample was taken from a Cisco router, and the ping successfully reached the destina-tion. But one could just as well use the ping command available in the command line ofthe host. We’re using a Windows host for our examples, but other platforms—such asMacs, the various UNIX platforms, IBM’s OS/400, and other proprietary server archi-tectures—all have ping and other basic network commands built into their operatingsystems.

Usually, the first ping test from a host is the link to its gateway router. On a com-puter running Windows XP, check this by clicking the Start button in the menu bar andchoosing All Programs | Accessories | Command Prompt to open a command promptwindow. Then check to see if the gateway router is responding by entering the ping command, as shown in the following code snippet:

Microsoft Windows XP [Version 5.1.2600]

(C) Copyright 1985–2001 Microsoft Corp.

C:\Documents and Settings\Tony>ping 10.1.13.1

Pinging 10.1.13.1 with 32 bytes of data:

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Ping statistics for 10.1.13.1:

Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Documents and Settings\Tony

The preceding example shows that four ping packets were sent to the gateway router,which failed to respond. This tells the troubleshooter a few things:

▼ The host PC’s NIC is good; otherwise, the operating system would havegenerated an error message when the card failed to respond to the ping command.

■ The Ethernet LAN segment might be down—a condition often referred to as a“media problem.” (The shared medium is apparently not working.)

▲ The network interface module on the gateway router might be faulty.

If the host checks out okay, the troubleshooter must move outward. As mentioned,the investigation should start with the link to the gateway router.






Extended Ping As useful and utilitarian as the ping command is for troubleshooting, itdoes have its limits. When using ping, the source address of the ping is the IP address ofthe interface that the packet uses as it exits the router. If you need more precision out ofyour ping, you can upgrade to the extended ping command.

Extended ping performs a more advanced check of your system’s ability to reach aparticular host. This command works only at the privileged EXEC command line, whereasa regular ping command works in both user EXEC and privileged EXEC modes.

Usage of extended ping on Cisco routers is fairly straightforward. Simply enter ping at the command prompt, and then press Enter . You will be prompted with a number ofconditions and variables. The default setting is enclosed in brackets. If you like the de-fault, simply press Enter ; otherwise, enter your preferred setting.

The following shows an example of an extended ping at work:

Router<>ping

Protocol [ip]:

Target IP address: 64.66.150.248

Repeat count [5]: 100

Datagram size [100]:

Timeout in seconds [2]:Extended commands [n]:

Sending 100, 100-byte ICMP Echos to 64.66.150.248, timeout is 2

seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (100/100), round-trip min/avg/max =

12/19/280 ms

NOTE The ping command also exists in i ts own form in Windows and UNIX/LINUX environments.Simply add the switches –s (UNIX/LINUX) or –t (Windows) after the ping command.

Using the show interfaces Command To check whether the problem is the gateway router’sinterface or the LAN segment’s medium, log into the gateway router, and enter the showinterfaces command to obtain the following report:

MyRouter#show interfacesEthernet1 is up, line protocol is down

Hardware is MyRouter, address is 0060.2fa3.fabd (bia 0060.2fa3.fabd)

Internet address is 10.1.13.1/28

.

.

.






In the preceding example, the router reports both that the Ethernet1 network inter-face module is up and the line protocol is down. The term line protocol denotes both thecable into the router and the LAN protocol running over it. A line protocol reportedas down probably indicates that the LAN segment’s shared medium—a hub, an accessswitch, or a cable—is faulty. From there, you would physically check the medium toidentify the hardware problem. (How to do that is covered later in this chapter in thesection “Troubleshooting Cisco Hardware.”)

Another potential condition could be that a network administrator has turned off theinterface or the line, or both. This is routinely done while a piece of equipment is beingrepaired, upgraded, or replaced. Notifying IOS that a piece of equipment is down formaintenance avoids having needless error messages generated by the router. The fol-lowing example shows the report when a network interface module is administrativelydown:

MyRouter#show interfaces

Ethernet1 is administratively down, line protocol is down

Hardware is MyRouter, address is 0060.2fa3.fabd (bia 0060.2fa3.fabd)


.

.

.

Whether a piece of equipment is down by design or because of a malfunction, it stillstops traffic. So it’s important to know when a piece of equipment is being worked on inorder to make sure an alternative path is available to handle traffic.

If both the gateway router interface and line protocol are up and running fine, thecause of the connectivity problem probably resides in a link to another network.

Troubleshooting Problems Connecting to Other Networks Things get a little more complicated beyond the home LAN segment. If the host can’tconnect beyond the gateway router, there are at once both more potential sources andmore types of trouble to check out. What’s meant by potential problem sources here is thatmany more hardware devices must be considered as potential causes of the reachabilityproblem. What’s meant by potential problem types is that such things as access lists, rout-ing protocols, and other factors beyond hardware must now also be considered.

Using the trace Command to Pinpoint Trouble Spots Instead of pinging outward from the

host one link at a time, the route between the host and the unreachable server can be analyzed all at once using the trace route command. In our example Windowshost, do this by choosing Start Run, and then type cmd to access the commandprompt. Once there, enter the tracert command, Microsoft’s version of the trace route command. The example in Figure 15-5 shows the route being traced from the host PC towww.PayrollServer.AcmeEnterprises.com, which is a fictional internal server severalhops away. It’s optional to use either the domain name or the IP address. Each line in thetracert command represents a hop along the path to the destination.






In TCP/IP internetworks, trace route commands work by sending three “trace”

packets to each router three times and recording the echo response times. As with theping command, the packets use the ICMP transport protocol. However, these packetsdiffer from ping packets in that they have a time-to-live (TTL) field used to incrementoutward from the host one step at a time. The TTL field causes the packet to die whenthe counter hits zero. The trace route command uses the TTL field by sending the firsttrace packet sent to the nearest router with a TTL of 1 to the next router with a TTL of2 and so on. This process is repeated until the destination host is reached—if it’s reach-able. The network administrator can put a limit on how many hops the trace may take toautomatically stop the process if the destination proves unreachable.

The ms readings are milliseconds, and you can see that nearby routers naturally tendto echo back faster. Under 10 ms is fast; anything over 100 ms or so is getting slow—butone must always adjust the timings according to how many hops removed the routeris. As you can see, the router in the shaded line in Figure 15-5 is the likely suspect forthe slow service because of its slow response times. The probable explanation is that therouter’s interface or the LAN segment attached to it is either congested or experiencinghardware faults. The next step would be to Telnet into router 10.1.49.12 (if possible)and diagnose the system, the involved network interface, and so on. If making a Telnet

Microsoft(R) Windows NT(TM)

(C) Copyright 1985-1996 Microsoft Corp.

C:\>tracert www.PayrollServer.AcmeEnterprises.com

Tracing route to www.PayrollServer.AcmeEnterprises.com [10.1.22.19]

over a maximum of 30 hops:

1 <10 ms <10 ms <10 ms 10.1.13.12

2 <10 ms 12 ms <10 ms 10.1.5.3

3 17 ms 20 ms 19 ms 10.1.17.22

4 22 ms 19 ms 23 ms 10.1.31.2

5 768 ms 831 ms 790 ms 10.1.49.12

6 31 ms 40 ms 42 ms 10.1.22.19

Trace complete.

C:\>

Host’s gateway router

Slow response indicatesthat this router is probablythe culprit.

Figure 15-5. The trace route command is a great way to pinpoint the source of a problem






connection isn’t possible, the troubleshooter must go in through the Console or AUXport, which, of course, requires that somebody be physically present at the device, unlessa dial-in maintenance solution has been configured beforehand.

Sometimes, a trace route will locate a node that’s stopping traffic altogether.An example of this is shown in Figure 15-6, where 10.1.49.12 now is dropping tracepackets instead of merely returning them slowly. The asterisks indicate a null timingresult because nothing came back, and the message “request timed out” is inserted.Take note that this does not necessarily mean the entire router is down. It could be thatonly the network interface or LAN segment that connects the suspect router may bedown or configured not to respond to pings.

If possible, first try to Telnet into the router through one of its other interfaces. If thisdoesn’t work, the next move depends on the router’s proximity. If it’s nearby, go to it andlog in through the Console or AUX port. If it’s remote, you should contact the personresponsible for dealing with it and walk that person through the diagnostic steps.

Figure 15-6. Here’s what happens if a traced route finds a router stopping traffic

Microsoft(R) Windows NT(TM)

(C) Copyright 1985-1996 Microsoft Corp.

C:\>tracert www.PayrollServer.AcmeEnterprises.com

Tracing route to www.PayrollServer.AcmeEnterprises.com [10.1.22.19]

over a maximum of 15 hops:

1 <10 ms <10 ms <10 ms 10.1.13.12

2 <10 ms 12 ms <10 ms 10.1.5.3

3 17 ms 20 ms 19 ms 10.1.17.22

4 22 ms 19 ms 23 ms 10.1.31.2

5 * * * request timed out











Trace complete.

C:\>

Host’s trace route commandis set to stop after 15 hops inthis example.

This time, thehost could notreach 10.1.49.12.

Each failed trace to 10.1.49.12 iscounted against the 15-hop limit.






NOTE Troubleshooting almost always takes place within the enterprise’s internetwork. This is because the network team can control events only within its autonomous system. The trace route command is a good example of this. If you traced a route through the Internet—say, to troubleshoot a VPN connection—many lines between your gateway router and the destination node will return asterisks instead of timings and “request timed out” messages instead of IP addresses. This is because almost all edge routers are configured by their network teams not to respond to trace routes.This is done as a security precaution. The point here is to highlight the trade-off a VPN must incur:

Loss of control is exchanged for very low-cost WAN links; you generally can’t troubleshoot somebody

else’s network.

Using the show interfaces Command Once the suspect network interface module has beenidentified, the troubleshooter must diagnose what’s causing the problem. The best wayto do that is to run the show interfaces command and review the latest statistics onthe interface’s operations. Remember, this information not only reflects on the interfacemodule itself, but also gives a rich set of clues as to what’s happening out on thenetwork.

An example show interfaces report is given in Figure 15-7. Don’t let its size and cryp-

tic terminology intimidate you. There is indeed a lot of information in it, but nothing thattakes a rocket scientist to understand.

Figure 15-7. The show interfaces command is one of the troubleshooter’s best tools

RemoteRouter#show interface

Ethernet1 is up, line protocol is up

Hardware is Lance, address is 0600.2fa3.faba (bia 0060.2fa3.faba)


MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255

Encapsulation ARPA, loopback not set, keepalive set (10 sec)

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:01, output hang never

Last clearing of "show interface" counters never

Queuing strategy: fifo

Output queue 0/40, 0 drops; input queue 0/75, 0 drops

5 minute input rate 43000 bits/sec, 1 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

2631684 packets input, 1135484504 bytes, 0 no buffer

Received 1508460 broadcasts, 0 runts, 0 giants, 0 throttles

2 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 input packets with dribble condition detected4508675 packets output, 444670670 bytes, 0 underruns

0 output errors, 3421 collisions, 1 interface resets

0 babbles, 0 late collision, 5778 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Interface status line MAC addressPrivate IP address

Subnet mask

Last fiveminutes

Totalinterfacetraffic

Interfaceconfig settingsand history

Ethernet

statistics


P Lib / Ci A B i ’ G id 4/E/V lt & V lt /6383 0/Ch t 15





This report is a snapshot of the interface at a particular instant in time. To check fortrends, the troubleshooter must run the show interfaces command intermittently to lookfor changes. The interface is identified by private IP address 10.1.49.12/28. Remember,usually only routers on the edge of an autonomous system—firewalls, Web servers,FTP servers, and the like—use public Internet addresses. The /28 notation lets otherrouters know that LAN segments attached to RemoteRouter are subnetted using the255.255.255.240 subnet mask. The notation uses 28 because the 255.255.255.240 mask has28 bits available for network addressing (as opposed to hosts). As mentioned earlier,mismatched subnets often cause problems.

The first thing to look at is the seventh line of the show interfaces report thatreads “Last clearing of show interfaces counters never” (highlighted in Figure 15-7).The example states that nobody has reset the report’s counters to zero since the lasttime the router was rebooted. The length of time since the statistics were last clearedis important, because most of the statistics are absolute numbers, not relative values,such as percentages. In other words, the longer IOS has been compiling the totals, theless weight the statistics should be given. For example, ten lost carriers in a day is a lot, but the same total over six months is not. To see when the last reboot was, use the showversion command, as shown here:

RemoteRouter#show version

Cisco Internetwork Operating System Software

IOS (tm) 4500 Software (C4500-IS-M), Version 11.2(17),

RELEASE SOFTWARE (fc1)

Copyright (c) 1986-1999 by Cisco Systems, Inc.

Compiled Mon 04-Jan-99 18:18 by etlevynot

Image text-base: 0x600088A0, data-base: 0x60604000

ROM: System Bootstrap, Version 5.3(10) [tamb 10],


BOOTFLASH: 4500 Bootstrap Software (C4500-BOOT-M), Version 10.3(10),


RemoteRouter uptime is 2 weeks, 3 days, 13 hours, 32 minutes

System restarted by power-on

.

.

.

The second-to-last line in the preceding example shows that the router has been upfor about two and a half weeks. Knowing this lets the troubleshooter more accurately judge whether certain error types are normal or excessive.

NOTE Historically, there have been a wide range of interoperability issues identified between different versions of IOS. When troubleshooting, one should make note of the IOS versions running in the environment and assess the impact of running different versions.


ProLib / Cisco: A Beginner’s Guide 4/E/Velte & Velte/6383 0/Chapter 15



ProLib / Cisco: A Beginner s Guide. 4/E/Velte & Velte/6383-0/Chapter 15


The exception to this sampling window is the two lines sitting in the middle ofFigure 15-7. These report input and output to the interface over the five minutes prior tothe report having been run. A troubleshooter trying to discern a trend in traffic patternswould periodically generate the show interfaces report and look at these numbers.

Statistics differ on what constitutes excessive. For example, Ethernet arbitratesmedia access control by collisions, so it’s normal for them to occur to some degree in ashared media environment—one with a hub, for example. The count of 3,421 collisionsin Figure 15-7 is okay for a period of two weeks or so, but a figure of 50,000 would in-dicate congested bandwidth. Broadcast packets are also normal, because they perform

positive functions, such as alerting routers of topology changes and providing otheruseful updates—again, within limits. There are over one and a half million in Figure 15-7,which might be excessive. However, what’s considered excessive is subject to so manyvariables that it must be left to the judgment of the troubleshooter. That’s where experi-ence comes into play. For a properly configured switched environment, there should bealmost no collisions on a single port. If you are seeing collisions, it’s quite possible thatyou have a speed or duplex mismatch.

Many statistics should ideally be low, or even at zero (depending on the time periodreported). For example, runts and giants are malformed packets sometimes caused by

a poorly functioning network interface card or an improperly configured VLAN. In aWAN link, lost carrier events probably indicate a dirty line or a failing telecommunica-tions component.

Table 15-4 defines many of the items reported using the show interfaces command.Knowing the items will help you understand how they can be used to diagnoseproblems.

Statistic Explanation

Five-minute rates(input or output)

The average number of bits and packets passingthrough the interface each second, as sampled overthe last five-minute interval.

Aborts Sudden termination of a message transmission’spackets.

Buffer failures Packets discarded for lack of available router buffer

memory.BW Bandwidth of the interface in kilobits per second

(Kbps). This can be used as a routing protocol metric.

Bytes Total number of bytes transmitted through theinterface.

Table 15-4. Definitions of Useful Ethernet Statistics


ProLib / Cisco: A Beginner’s Guide 4/E/Velte & Velte/6383-0/Chapter 15



ProLib / Cisco: A Beginner s Guide. 4/E/Velte & Velte/6383 0/Chapter 15



Carrier transitions A carrier is the electromagnetic signal modulated bydata transmissions over serial lines (like the soundyour modem makes). Carrier transitions are eventswhere the signal is interrupted, often caused whenthe remote NIC resets.

Collisions The number of messages retransmitted due to an

Ethernet collision.

CRC Cyclic redundancy check, a common techniquefor detecting transmission errors. CRC works bydividing the size of a frame’s contents by a primenumber and comparing the remainder with thatstored in the frame by the sending node.

DLY Delay of the interface’s response time, measured inmicroseconds (µs), not milliseconds (ms).

Dribble conditions Frames that are slightly too long, but are stillprocessed by the interface.

Drops The number of packets discarded for lack of spacein the queue.

Encapsulation The encapsulation method assigned to an interface(if any). Works by wrapping data in the header ofa protocol to “tunnel” otherwise incompatible datathrough a foreign network. For example, Cisco’sInter-Switch Link (ISL) encapsulates frames from

many protocols.Errors (inputor output)

A condition in which it is discovered that atransmission does not match what’s expected,usually having to do with the size of a frame orpacket. Errors are detected using various techniquessuch as CRC.

Frame The number of packets having a CRC errorand a partial frame size. Usually indicates amalfunctioning Ethernet device.

Giants Packets larger than the LAN technology’s maximumpacket size—1,518 bytes or more in Ethernetnetworks. All giant packets are discarded.

Table 15-4. Definitions of Useful Ethernet Statistics (continued)





g



Ignored Number of packets discarded by the interfacefor lack of available interface buffer memory (asopposed to router buffer memory).

Interface resets When the interface clears itself of all packets andstarts anew. Resets usually occur when it takes toolong for expected packets to be transmitted by the

sending node.Keepalives Messages sent by one network device to another to

notify it that the virtual circuit between them is stillactive.

Last input or output Hours, minutes, and seconds since the last packetwas successfully transmitted or received by theinterface. A good tool for determining when thetrouble started.

Load The load on the interface as a fraction of thenumber 255. For example, 64/255 is a 25 percentload. This counter can be used as a routing protocolmetric.

Loopback Whether loopback is set to on. Loopback is wheresignals are sent from the interface and thendirected back toward it from some point alongthe communications path; used to test the link’susability.

MTU The maximum transmission unit for packetspassing through the interface, expressed in bytes.

Output hang How long since the interface was last reset. Takesits name from the fact that the interface “hangs” because a transmission takes too long.

Overruns The number of times the router interfaceoverwhelmed the receiving node by sending morepackets than the node’s buffers could handle.It takes its name from the fact that the routerinterface “overran” the sender.

Queues (input andoutput)

Number of packets in the queue. The number behind the slash is the queue’s maximum size.

Queuing strategy FIFO stands for “first in, first out,” which meansthe router handles packets in that order. LIFOstands for “last in, first out.” FIFO is the default.







Now that we’re introduced to the various statistics compiled in the show interfaces report, let’s review how to read it. Figure 15-8 shows the Ethernet statistics portion of thereport, this time, with some of the more important variables highlighted. These are thevariables an experienced network administrator would scan first for clues.

More often than not, connectivity problems are caused by some type of configurationproblem, not by a piece of failing equipment. Depending on the Ethernet statistic thatis high, the interface may be overwhelmed by incoming traffic, have insufficient queuesize configured, have insufficient buffer memory, or be mismatched with the speed of anetwork sending input.

Checking Access Lists for Proper Configuration The classic example of a devicemalfunctioning even though its hardware is running fine is the misconfigured access list.You’ll recall that access lists are used to restrict what traffic may pass through a router’sinterface, thereby cutting off access to the LAN segment attached to it. The access listdoes this by inspecting for source and destination IP addresses—a way of controllingwho may go where. The extended access list also uses port numbers to further restrictwhich applications may be run once you’re admitted. Indeed, access lists are the most


Rely The reliability of the interface as a fraction of thenumber 255. For example, 255/255 is 100-percentreliability. This counter can be used as a routingprotocol metric.

Runts Packets smaller than the LAN technology’sminimum packet size—64 bytes or less in Ethernet

networks. All runt packets are discarded.

Throttles The number of times the interface advised a sendingNIC that it was being overwhelmed by packets being sent and to slow the pace of delivery. It takesits name from the fact that the interface asks the NICto “throttle” back.

Underruns The number of times the sending nodeoverwhelmed the interface by sending more packets

than the buffers could handle. Takes its name fromthe fact that the router interface “underran” thesender.









permit icmp any 209.98.208.32 0.0.0.15 echo-reply

.

.

.

Looking at the preceding example, access list 100 explicitly denies traffic to a certainIP address. This is frequently done to stop outbound traffic to a known undesirable IPaddress or some other type of router that could allow hackers a crack at the enterprise’sedge router. Access list 101 is more sophisticated, with a series of permit rules to control

which applications may be used between hosts. The application’s IP port is defined behind each eq modifier, such as eq smtp for e-mail or eq 65 for TACACS+ databaseservice. (Certain ports can be identified by an acronym; others must be identified by anumber.) Also note that access list 101 has only permit rules. This is possible because ifa packet’s request for service isn’t explicitly permitted, it will be denied by the “implicitdeny” rule when it reaches the bottom of the access list.

It could be that the inadvertent deny rule, lack of a permit rule, or simple typois causing the problem. The troubleshooter would scan the access lists for any rulesthat might be causing the problem at hand. For example, if a person can’t connect to

the mail server, the troubleshooter would look for statements containing eq smtp or the mail server’s IP address. The next step would be to go to the interface connectingthe network experiencing the problem to see if the access-group command was usedto apply the questionable access list to it. To do this, you must enter privileged EXECmode and go into configure interface mode pointed to the interface in question, asshown here:

MyRouter#enable

Password:

MyRouter#show running-config

.

.

.

interface Ethernet1

ip address 10.1.13.1 255.255.255.240

ip access-group 100 in

ip access-group 101 out

.

.

.

If the questionable access list is in force, double-check that the access list is beingapplied in the correct direction for the interface. If that checks out, temporarily disable itto see if traffic can pass the router without it. There are two access lists in our example, sowe would disable them both to see if the problem is being caused by access lists. Disableaccess lists on the interface as follows:










Serial links have an obvious importance because they extend internetworks beyondthe office campus to remote locations. A remote link of any size requires using a digitaltelephone circuit of some kind, ranging from a fractional T1 up to a full T3 (DS3) line.

CSU

DSU

Serial

line

Serial interface

Router

A serial line provides a window through which its entire WAN link can be diagnosed.In other words, not only can you analyze the serial line and its interfaces, but by lookingat the traffic it carries, you can also diagnose the digital phone loop and, to some extent,what’s happening at the remote end of the link.

Differences in the show interfaces serial Report Cisco provides a special tool for troubleshooting serial links in the show interfaces serial

command. It’s largely the same as the normal show interfaces command, but with someimportant differences, as highlighted in Figure 15-9. Specifically, it shows informationfor the serial port.

One way serial links differ is the type of encapsulation used over digital telephoneloops. The High-Level Data-Link Control (HDLC) encapsulation protocol is indicatedin the top shaded box in Figure 15-9. Encapsulation is necessary to maintain Ethernetpackets over the digital telephone link. Sometimes, encapsulation may have beeninadvertently turned off, so the Encapsulation field should be selected.

Another difference is that conversations (sessions) are reported in the show interfaces

serial interface_number report. WAN links have less bandwidth than local shared media. Towit, a T1 (DS1) circuit has a data rate of 1.544 Mbps, and a T3 (DS3) has a rate of 45 Mbps.Most enterprises use fractional T1 or T3 by purchasing channels within them (T1 has24 channels; T3 has 672). WAN bandwidth, therefore, is limited compared to, say, a100-Mbps LAN segment, and sometimes a particular user session takes more than its share.Therefore, when troubleshooting a WAN link, it helps to know how many conversationsare going on. In case you’re wondering, the Reserved Conversation field has to do with theResource Reservation Protocol (dubbed RSVP). RSVP is an industry standard designed foruse in QoS (Quality of Service) tools to help guarantee service levels.

The box at the bottom of the figure shows a third difference in the show interfacesserial interface_numberreport. These five fields are the same as the blinking lights you mayhave noticed on external modems. For example, DTR stands for Data Terminal Ready,an EIA/TIA-232 (née RS-232) circuit that is activated to notify the data communicationsequipment at the other end that the host is ready to send and receive data. DCD standsfor Data Carrier Detect, which is important because it senses the actual carrier signal (themodem noise you hear when making a modem connection). The five modem circuits areincluded in the show interfaces serial interface_number report for troubleshooting seriallinks that run over analog/modem lines instead of digital lines.






Key Diagnostic Fields in the show interfaces serial Report Serial links differ by nature from LAN segments, so diagnosing them takes a differentfocus. Certain things that are, to some extent, taken for granted in LAN segment linksare often the cause of performance problems or even failures in serial links. Figure 15-10highlights the items that troubleshooters look at first in a serial interface.

As you can see, troubleshooting serial links emphasizes looking at errors and lineactivity. This is natural, given that the middle part of a WAN link—the telephonecircuit—is basically invisible to networking equipment.

Looking at Figure 15-10, we see a case in which input traffic seems to be going okay,

but a lot of output packets are being dropped. Given that the serial line is being pushedhard, running at about 80 percent of available bandwidth, we can conclude that thedrops are being caused by overuse, not by faulty hardware in the link.

The first line of output in Figure 15-10 can also help your troubleshooting efforts. Inthe show interfaces serial display, the first line will give one of five status indications.Ideally, as Figure 15-10 shows, you want this line to read “Serial x is up, line protocol isup.” However, if there is a problem, that status might be one of the following:

Figure 15-9. Most WAN links still use serial lines to connect routers to phone loops

RemoteRouter>show interface serial0

Serial0 is up, line protocol is up

Hardware is HD64570



Encapsulation HDLC, loopback not set, keepalive set (10 sec)



Input queue: 0/75/390 (size/max/drops); Total output drops: 54920

Queueing strategy: weighted fairOutput queue: 0/1000/64/12921 (size/max total/threshold/drops)

Conversations 0/1/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)






4783008 packets output, 2510565558 bytes, 0 underruns



12 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up






▼ Serial x is down, line protocol is down. This is an indication that the router

is not sensing a signal from the WAN connection, there is a problem with thecabling, or even a problem at the telephone company.

■ Serial x is up, line protocol is down. This is an indication that a local orremote router has been misconfigured, keepalives are not being transmitted bythe remote router, or local or remote channel service unit or digital service unitshave failed.

■ Serial x is up, line protocol is up (looped). This is an indication that there isa loop in the circuit.

■ Serial x is up, line protocol is down (disabled). This is an indication thatthere is a high error rate because of a problem with the telephone carrier,channel service unit or digital service units are experiencing a problem, or therouter interface is faulty.

▲ Serial x is administratively down, line protocol is down. This is anindication that the router configuration includes the shutdown commandor that a duplicate IP address exists.

Figure 15-10. Certain fields are usually the focus when troubleshooting a serial link

RemoteRouter>show interface serial0

Serial0 is up, line protocol is up

Hardware is HD64570



Encapsulation HDLC, loopback not set, keepalive set (10 sec)



Input queue: 0/75/390 (size/max/drops); Total output drops: 54920

Queueing strategy: weighted fair

Output queue: 0/1000/64/54920 (size/max total/threshold/drops)

Conversations 0/1/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)






4783008 packets output, 2510565558 bytes, 0 underruns



12 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

Serial line load percentageis high, around 80%.

High numbers ofoutput drops andresets suggest lineis overutilized

Input error count is relatively low.






Troubleshooting Serial-Line Input Errors One of the most common causes of serial-lineproblems is input errors—in other words, data inbound from the remote site. Probablecauses of serial-line input errors, with suggested actions, are outlined in Table 15-5.

Troubleshooting Serial-Line Input and Output Errors Another clue to serial-line problems isan increase in dropped packets at the interface. A drop occurs when too many packetsare being processed in the system and insufficient buffer memory is available to handlethe packet. This applies to both input and output drops, as outlined in Table 15-6.

Drops taking place in one direction but not the other (input versus output) canpoint the troubleshooter toward the problem’s source. If they’re happening both ways,the router or its serial interface is probably the culprit.

Input Error Symptoms Probable Causes and Suggested Actions

Input errors along

with CRC or frame errors

A dirty line, where electrical noise interferes

with the data signal. Serial cable exceedsmaximum length specified for the type ofphone circuit. Serial cable is unshielded. Thephone circuit itself may be malfunctioning.

Actions: Reduce cable length. Install shieldedcable. Check phone loop with a line analyzer.

Clocking jitter in line where data signal variesfrom reference timing positions, or clockingskew where device clocks are set differently.

Actions: Make sure all devices are configuredto use a common-line clock.

Input errors alongwith aborts

The transfer of a packet terminated inmidtransmission. Usually caused by aninterface reset on the router being analyzed.Can also be caused by a reset on the remoterouter, a bad phone circuit, or a bad CSU/DSU.

Action: Check local hardware, then remotehardware. Replace faulty equipment.

Table 15-5. Input Errors Causes and Actions






Troubleshooting Serial Links Most of us have used modems long enough to know thatsometimes an established connection can falter, or even be broken. This goes for serial lines,too, usually because of interface resets or carrier transitions, as outlined in Table 15-7.

Although they’re not LAN segments per se, serial links are integral to geographicallydistributed internetworks. Don’t forget to consider them, even when a serial-line problemis not initially apparent. For example, when evaluating performance problems, it could be that a faulty serial link is shifting traffic loads elsewhere within the internetwork.

Client-Server VPNs As we’ve discussed already, VPNs are a cost-effective way to use the Internet as yourown private WAN. If you’re having trouble getting a VPN to work, there are four areasin which VPN problems generally fall:

▼ Blocked VPN traffic

■ Bad Internet connections

■ Configuration errors

▲ Network Address Translation (NAT) tunneling problems

Packet Drop Symptoms Probable Causes and Suggested Actions

Increase in droppedinput packets

Input drops usually occur when traffic is being routed from a local interface (Ethernet,Token Ring, FDDI) that is faster than the serialinterface. The problem usually emerges duringperiods of high traffic.

Actions: Increase the interface’s input hold

queue size in the router’s config file.Increase in droppedoutput packets

Output drops happen when no system buffer isavailable at the time the router is attempting tohand the packet off to the transmit buffer duringhigh traffic.

Actions: Increase the interface’s output holdqueue size. Turn off fast switching. Implementpriority queuing.

Table 15-6. Dropped Packet Causes and Actions






At the risk of insulting anyone’s intelligence, when problems arise (not only VPNissues), the first thing to do is to check for loose cables. Wiggle the cables on the client’smodem, the router, and firewall to ensure they are seated properly. It’s also a good ideato make sure you’re using straight-through Cat 6 or 7 cabling and didn’t pick up a lengthof crossover cable.

Blocked Traffic The next step is to ensure your Internet service provider (ISP) allows IPSec VPN traffic.

If your provider does not, it will not matter if your VPN is properly configured, becausethe packets won’t be going anywhere. If your ISP does not allow IPSec VPN traffic, youmight have to consider changing ISPs.

Check your firewall to ensure that it isn’t blocking IPSec or PPTP traffic. To make aVPN connection, it is necessary to configure outbound IPSec traffic on the firewall. To dothis, you must configure your firewall to enable IPSec, and then create a rule allowing thepassage of traffic between the LAN and WAN. If that’s not possible, it might be necessaryto locate the client in the DMZ or consider investing in a different router or firewall.

Line Error Symptoms Probable Causes and Suggested Actions

Increasing carriertransitions

Interruption in the carrier signal. Usually due tointerface resets at the remote end of the link. Resetscan be caused by external sources such as electricalstorms, T1 or T3 overuse alerts, or faulty hardware.

Actions: Use breakout box or serial analyzer tocheck hardware at both ends. Then check router

hardware. Replace faulty hardware as necessary.No action required if problem was due to externalcause.

Increasing interfaceresets

Interface resets result from missed keepalivemessages. They usually result from carriertransitions, lack of buffer, or a problem with CSU/DSU hardware. Coincidence with increased carriertransitions or input errors indicates a bad link or bad CSU/DSU hardware.

Actions: Use breakout box or serial analyzer tocheck hardware at both ends. Contact leased-linevendor if hardware is okay.

Table 15-7. Serial Line Error Causes and Actions






Not only can hardware firewalls block traffic, but so can software firewalls. This isanother easy place to check, especially for clients that are traveling or trying to connectfrom locations that aren’t equipped with hardware firewalls, but that are set up withsoftware firewalls. Just disable the software firewall and see if that works. Some softwarefirewalls will ask you if they should allow VPN traffic to be passed and you can add thedesired destination IP address to the trusted-zone setting.

NAT Make sure your NAT is tunneling correctly. A good place to start is by making certain

you have the most current firmware updates and software. When IPSec tries to verify thepackets’ integrity, NAT changes the source IP address to the firewall’s WAN address toproperly navigate the Internet. Unfortunately, this causes problems with IPSec becausethe packets fail an integrity check.

You can get a listing of your NAT translations and an overview of your NAT statistics by using two simple EXEC commands:

▼ show ip nat translations verbose This displays the active NAT translationswith additional information for each translation table, including how long the

entry has been used.▲ show ip nat statistics This displays a variety of NAT statistics, including the

number of active translations, interfaces, and total translations.

Configuration Configuration can also be the culprit when trying to track down VPN problems. Ensurethat the correct IP addresses are being used. For client VPNs, checking and renewingthe IP address in Windows is accomplished by opening a command prompt and then

entering ipconfig /all.If the IP address issued by the network administrator to connect to the VPN does

not fall within the range shown, then the IP address is not valid. To correct this, renewthe lease. This is accomplished by opening a command prompt window and typingipconfig/renew and the IP address of the adapter.

NOTE If the client is using PPPoE to connect to the ISP—which will be the case if a static IP

address has not been assigned—make sure the client is connecting to the Internet using whatever connection application is needed.

Send a ping command to your VPN server’s IP address. If you get a response, thenyou know the client is connected to the Internet and able to see the VPN server. Next,you should rule out any DNS configuration problems. This time, conduct a ping test, butuse the domain name (for example, www.velte.com). If you get a response, the Internetconnection is working fine. If not, it means DNS is misconfigured either at the client oron the DNS server itself.






The client and the VPN server must be able to speak the same language to get the jobdone. As such, it’s important to make sure that the encryption settings on both the clientand VPN server are the same. Authentication algorithms must be configured properlyon both the client and VPN server. Both devices will need the shared secret, or, if usingcertificates, the correct public key is necessary.

Bad Connections Next, check whether the client is trying to connect over a slow connection. Latency cancause VPN connections to fail, because they like consistent traffic; otherwise, they tend

to drop off. You’re most likely to see this as an issue with satellite connections wherelatency can run from half a second to several seconds.

Connection speed can be checked with the ping tool. Using the “-t” switch, you canget a continuous test of connection speeds between the client and the VPN server. Forinstance:

ping 68.93.44.123 –t

This produces a list of the test’s efforts to send packets to the address. The test isended by pressing CTRL-C. Take a look at the results. If you see any stray “Request timed

out” error messages, try increasing the timeout value so that you can accurately gaugehow much latency your connection suffers. This value can be changed by using the “-w”switch. For instance:

ping 68.93.44.123 –t-w 7000

This increases the timeout value to 7,000 ms. This should be enough to indicate howmuch latency is present on your link. Connection times at 1,500 ms and above will causethe VPN link to fail.

NOTE It might not sound important at first blush, but if the VPN server and client are not in the correct time zones and have the correct time settings, they might not be able to hook up. This is because correct time settings are necessary for key expiration.

TROUBLESHOOTING CISCO HARDWARE When the likely location of the problem has been identified, the first step is to physicallyexamine and test the suspect device. This will identify the problem’s cause in a surprising

number of troubleshooting situations. Sometimes, the problem is caused by somethingas simple as a loose component; other times, something is damaged.

Inspecting Devices Once a suspect device is identified, it should be physically inspected. This is routineprocedure. (Even when a suspect or troubled device is in a remote location, a contactperson is sent to make an inspection.) Earlier, we stated that most troubleshooting tasks






are done from the administrator’s desk, and that’s true. Most tasks are done from theadministrator’s PC or an NMS console. However, there’s no substitute for actuallylooking at a device to see what’s going on. The two parts of inspecting a device arereading its LEDs and inspecting the device’s components.

Reading Device LEDs If the device is still online, the first thing to do is to read the LEDs (light-emitting diodes).You probably recognize LEDs as those blinking lights on the front of many electronicdevices. Virtually all network devices have LEDs to assist in troubleshooting. The LED

bank arrangement follows the device layout:

▼ Access devices with a bank of ports on the front, with a twisted-pair cableplugged into each port using an RJ-45 phone-style jack . Products from theCisco Catalyst 2800 to the Catalyst 4500 Switch fit this description. There isusually one LED per port.

■ Motherboard-based routers with LAN segments plugged into the back,usually through twisted-pair cable, but also fiber-optic cables for uplinks . The Cisco 7200 Router fits this description. LEDs on these devices appear

behind smoked-plastic panels on the front of these boxes. ▲ High-end routers and switches of the bus-and-blade configuration, again

with networks plugged into the back, both fiber-optic and twisted-paircable. The Cisco 7500 Router and Catalyst 6500 Switch fit this description.LEDs on these devices appear both behind smoked-plastic panels on the frontand on the blades (card modules) themselves on the back (remember, a blade is basically an entire router or switch on a board).

LEDs are also called activity lights. Each LED on an access device represents a host.

Router and LAN switch LEDs represent entire LAN segments.LEDs blink and change colors according to the port’s status. Green means okay, and

orange means the port is coming up. If the port is down, its LED goes dark. The port’sLED blinks when packets are passing through it. A common practice is to press Reset tosee what happens. LEDs temporarily go orange or even red if they encounter troubleduring the power cycle. They will eventually go green, but the temporary error condi-tion may indicate a nonfatal configuration error.

The rule is that if an activity light is green, the line is good and the problem muststem from some type of configuration problem. If the light is orange, the line is operating

but malfunctioning. If the activity light is off, the line is down.

Physically Inspecting Devices The next step is to physically inspect the device itself. Start by making sure the deviceis offline, and then remove the cover from the top of the device chassis and inspect theinterior, looking for the following:






▼ Loose connections Look for any loosely attached card (module) or cable.Reseat any that are found.

■ New cards If you know any card to be new, reseat it into its connectionseveral times. New cards are more prone to oxidation or carbon film buildupon their backplane connections.

■ Burned or damaged parts Look for any burned wires, ribbon cables, or cards.Also look at the backplane to see if it’s okay. Closely inspect the wires leadingto the device’s power supply. Also look for any crimped wires.

▲ Dirty device interior If the device has dust and lint in the interior, turn off thedevice and clean it. Devices can accumulate a lot of foreign substances from theair in dusty or dirty environments, which sometimes can affect performance.

After completing the inspection, try rebooting the device to see if power-cycling itwill fix the problem. One important caution: Don’t change anything in the configura-tion. Doing so before rebooting can make it difficult to determine the problem’s sourceafterward; it only adds more variables to the mix.

The Reboot Test If no severe problem was found inspecting the device, the next step is to try a power-cycle test to see how it responds. Power-cycle means to turn a device off and then turn iton again, which you probably know as the cure-all for Microsoft Windows. As we saw inChapter 3, rebooting devices can tell a lot about the status of a device, and, in some cases,it even makes the problem go away.

When you reboot, if the configuration in memory is mismatched with the hardware, avariety of problems can ensue. Ports might hang, bus timeout errors may occur, and so on.If the device reboots and prompts for a password, the circuitry and memory are working

properly. Some major symptoms and probable causes are outlined in Table 15-8.When hardware problems this extreme are encountered, it’s time to call in support

from Cisco or a third-party maintenance organization with which your enterprise hascontracted. Typically, devices are shipped into the maintenance center for bench repair.Only end-user enterprises with spare parts, Cisco-trained personnel, and proper instru-ments attempt to repair networking hardware devices in-house.

TROUBLESHOOTING NETWORK CONFIGURATIONS As your network grows and evolves, you’ll likely encounter some LAN segments thathave wireless capabilities—and their own set of problems. In addition, you’ll more thanlikely want to track down issues related to your network’s overall performance. In thissection, let’s take a closer look at how to track down and resolve problems with a wirelessnetwork, as well as some good methods for locating and fixing problems stemming fromperformance issues.






Wireless Networks Wireless networks provide a whole new level of convenience to the world of networking.The ability to connect computers without having to worry about wiring—not to mentionthe ability to take your laptop anywhere in the office—and still maintain network con-nectivity is a huge plus.

Many wireless deployments fire up as soon as you connect your access point andwireless card. For example, if you’re using a Plug-and-Play–capable version of Windows

(like Windows XP, for example), most times, the wireless card will be instantly recog-nized and, security issues aside, you’ll have access to the network with no problems.Regrettably, it doesn’t always work that smoothly.

Address Filtering Like so many facets in networking, what is meant to keep the bad people out can alsokeep the good guys out. One of WiFi’s methods of security is MAC and IP address filter-ing. That is, your APs (access points) can be configured to allow or reject traffic from spe-cific IP or MAC addresses. If a client or clients are having trouble connecting, make sure

their IP or MAC addresses are not being blocked by the AP. When properly configured,however, this feature is a nice way to add a layer of security.

Channel Interference In an 802.11b/g network, there are really only three channels that are usable: channels1, 6, and 11. If you’re having trouble with your network, you might check to see if therearen’t other networks already existing on one of those channels. You may be experiencing

Reboot Symptom Probable Causes

No response Bad power supply; blown fuse; bad breaker; badpower switch; bad backplane

Won’t reboot Bad or miswired power supply; bad (or poorlyseated) processor card; bad memory board; badIOS image in NVRAM; shorted wires

Partial or constant

reboot

Bad processor, controller, or interface card; bad

backplane; bad power supply; bad microcode

No cards show up in boot display

Bad processor, controller, or interface card; bad backplane; cards not seated in backplane; badpower supply

Table 15-8. Typical Reboot Problems and Probable Causes






interference from another device on your own network, or you might be experiencingcomplications from a neighbor’s wireless network. The solution to your problem might be as simple as changing the AP and clients’ channel.

Encryption The ideal means of security for your wireless network is to employ 802.1x authentica-tion along with encryption. However, if you aren’t using authentication, you shouldat least use encryption. An easily overlooked component of wireless networking is en-abling encryption on your access points and clients. As we mentioned in Chapter 8, this

is extremely important for the sake of protecting your data.If encryption is not enabled, it is relatively easy for someone to sniff the wireless net-

work traffic and glean all sorts of information, from user ID and password informationto the contents of e-mails being sent and received.

Even though it has been found to be quite insecure, Wireless Equivalent Privacy(WEP) is the still most common encryption protocol in use today. It uses a key that youestablish on the access point and then enter into your wireless-enabled devices. This keyis used to encrypt the data being transmitted and decrypt incoming data. Without thekey, no one else can “see” the data as it is transmitted.

The next level of wireless security is WiFi Protected Access (WPA). It’s like WEP, butprovides a much higher level of encryption and authentication security. It’s not availableon older access points, and it may take an upgrade on the client side to enable it there. Ifyou can’t use WPA, at least run WEP—some encryption, even if WEP has been cracked,is better than none at all.

NOTE It is wise to become familiar with the encryption and overall security features available on

your hardware and software—the standards and technology are advancing quickly in the wireless arena, and sometimes, a more secure environment is only a firmware or software update away.

WEP With encryption, however, comes its share of problems. If your encryption schemeis incorrectly set on either the access point or clients, then expect problems.

If you’re having problems getting your wireless clients to connect, the first place tostart is by checking encryption settings. Follow these steps to ensure your encryptionsettings are correct:

1. Turn off encryption. Even though we just said it was important to have

encryption enabled, if you turn off encryption and are still having problems,then you know encryption isn’t to blame. If it turns out that everything isrunning fine, move on to step 2.

2. Count characters. Check your access point and WiFi card instructions to makesure you’re entering the correct number of characters for the encryption key. Forinstance, when using a 40-bit WEP key, Cisco Aironet 350 requires five ASCII orten hexadecimal characters for its encryption key (this is shown in Figure 15-11).






Also, check to see if you must specify whether you are using an ASCII stringor hexadecimal string for the key. Table 15-9 shows how many characters areneeded for various bit-lengths of keys.

3. Configure authentication methods. When using WiFi, there are two typesof authentication employed: open system and shared key. Reconfigure the accesspoint and client to allow open system, thus disabling WEP. When you enableWEP, change over to the shared key authentication. This provides optimalsecurity.

Figure 15-11. Mistyping a character in the WEP can cause WiFi networks to fail

WEP Bit Levels ASCII Hexadecimal40/64 5 characters 10 characters

128 13 characters 26 characters

Table 15-9. The Number of Characters Needed in Various Key Lengths






4. Match your WEP levels. Although it’s possible to mix environments in which40/64-bit and 128-bit devices are operating (we’ll talk about that more in amoment), it’s best to make sure everyone is using the same level. That said, if itturns out you need to work in a mixed environment, 128-bit devices can talk to40/64-bit WEP devices only if they are set to use 40-bit keys.

5. Check your passphrases. Some WiFi vendors (including Cisco) allow you toenter passphrases for key generation. That is, you don’t have to come up witha string of hexadecimal characters. If you like, you can come up with a simplephrase. (For instance, Figure 15-12 shows the passphrase “chunkymonkey”

turned into a hexadecimal WEP key.) This is a convenient tool, because whensetting up the key, you don’t need to remember a series of meaningless lettersand numbers—“chunkymonkey” is easier to remember than “63B27312BB.”When you use passphrases, there are a couple things you should keep in mind.First, keep the passphrase string short. You don’t need to come up with phraseslike “supercalifragilisticexpialidocious”—it won’t result in a key that is anymore secure than one generated with a shorter passphrase. Second, use lettersand numbers only—don’t throw spaces, punctuation, or other symbols into themix.

WPA If you’re using WPA or WPA2, there are a lot of finicky little steps in the configurationthat could lead to misconfiguration.

Look back on the WPA configuration we outlined in Chapter 8. Here are some sourcesof problems that might be possibilities:

Figure 15-12. Passphrases can be used to generate WEP keys






▼ With WPA, the Cipher option must be selected and TKIP chosen from thedrop-down menu.

■ WPA requires that the encryption key be entered in key number 2, not keynumber 1. Ensure that this has been properly set.

■ The correct SSID must also be selected. This setting is made using the SSIDManager and then selecting the correct SSID from the Current SSID List.

▲ You might also be experiencing problems depending on the authenticationmethod you’ve chosen (or need to choose). The authentication method, which

is also set from the SSID Manager screen, should be set based on which typeof clients your WiFi network is using. If you’ve only got Cisco clients, selectNetwork-EAP. If you’re using third-party clients, select Open Authenticationwith EAP. If you’re in a mixed environment with both Cisco and third-partyclients, select both Network-EAP and Open Authentication with EAP.

Also, take a look back on the suggestions for WEP. Since WPA and WPA2 utilize keys,check such things as key lengths and ensure that the keys have been properly entered on both the AP and the clients.

Antenna Placement and Interference With a wired network, you don’t need to worry too much about interference from otherdevices. For instance, running the photocopier probably won’t cause any trouble withyour wired workstations, but curiously, it may cause your wireless connection to drop out.And even though your wireless-enabled laptop affords you the freedom to go anywherein your office, you can only be from 100 to about 300 feet from your access point. Afterthat, interference from walls, floors, and other obstructions will cause connections toslow appreciably or drop out altogether. Of course, this still beats the pants off a wiredconnection, which only lets you roam as far as the Cat 5 tether allows, which might

be no further than one corner of your desk. Wireless networking can be worth doing, just remember to keep in mind where your wireless devices will be in relation to anaccess point. In most cases, try and locate your access point as centrally as possible tothe clients.

No matter where you place your access points, always be aware of sources ofinterference. It was mentioned earlier that photocopiers have been known to reduceconnectivity in WiFi networks, but be mindful of other devices that can wreak havoc onyour system. A main culprit comes in the guise of the 2.4-GHz cordless telephone. Sincethis operates on the same frequency as 802.11b/g, it can cause some headaches. If you

suspect a cordless phone or other 2.4-GHz device, try using other WiFi channels to seeif things improve.

Extending Your Wireless Network’s Range What if you just can’t get a good signal in someareas of your space and you really want wireless there? After ruling out interferencefrom another device, repositioning your antenna(s), and perhaps relocating your accesspoint, you may just want to buy an additional access point. This extra access point can beused to extend the range of your wireless network, as Figure 15-13 shows.






When using an access point to extend range, you can do so without needing a wiredconnection by configuring the access point as a bridge from an existing access point. Just make sure you are monitoring performance and capacity as your user count grows, because the wired access point could become saturated with network traffic and becomea network bottleneck.

Checking Your Levels A simple way to check your connectivity levels is to start the clientin the same room or location as the access point. When you’ve got the two devicescommunicating, it’s easy enough to start moving the client away from the access point.This will give you a quick and dirty idea of the range between the two.

However, you can plot your devices’ connectivity with a little more finesse by usingthe Cisco Aironet Client Utility. Once started on your client, this application, shown inFigure 15-14, shows the quality and strength of your wireless signal.

Point-to-Point Troubleshooting If your wireless bridge link stops working, it is possiblethat there is a problem with your system’s antennas, cabling, or connectors. Check your

antennas and ensure they have not come out of alignment.Also, antennas and connections can be damaged by moisture. If the antennas are

not sealed properly when they’re installed, moisture can condense inside the antennafeedhorns, ultimately filling them with water. Moisture that makes its way into coaxialcabling can be even more problematic. Coax cables have a foam internal dielectric. Thiscan act like a sponge, sending moisture along the length of the cable.

Figure 15-13. It’s possible to extend your wireless network’s range with additional access points

Wireless-enabled PCThe device is out of the accesspoint’s range.

Aironet Access Point

Aironet Access Point Additional access pointsextend the range of yourwireless network.

Wireless-enabled PC






NOTE If you determine that coax cabling has been compromised and is sucking up moisture,replace the entire length, rather than snipping off a few feet and replacing the connector.

When problems manifest themselves in outdoor systems, the effect will appear on bothends of the link to the same degree. This is relevant to know, because if you see a degradedsignal on one end of your link, don’t automatically think you’ve found the location of theproblem. It might very well be on the other side of the link. Check both ends.

On the other hand, if the receive-signal is low on one end but not the other, generally,

this is a problem caused by misconfiguration of the radio units or by interference. Assuch, don’t make a bad situation worse by realigning antennas. If you determine that thesetup is correct and the equipment is working properly, check for anything that mightcause interference before adjusting the antennas.

If you suspect interference as the culprit, examine your system and its behavior. Is theproblem continuous, or is it intermittent? Most often, interference occurs intermittently,when the source of interference becomes active.

Figure 15-14. The Cisco Aironet Client Utility shows your signal strength and quality



659




For point-to-point wireless networks, determining the source of interference can bea horrendous chore. First, look around the antennas at each end of your link. Are thereany other antennas present? If so, do a little sleuthing to determine who owns it, whooperates it, at which frequency it operates, how much power it is transmitting, and whattype of antenna polarization is being used.

Once you’ve tracked down this data (it could be just as simple as asking around inthe building on which the antenna is mounted), the next step is to ask the owner if he orshe would be willing to help you determine if their system is the source of your system’sinterference.

When you have all the pertinent information about the interfering source, you canmuch more easily resolve the problem. First, consider your own antennas. Are any ofthem pointed at the other system’s antennas? Is it possible to reposition your antennasso they are out of the other system’s broadcast path?

Often, changing the polarization of your antennas to the opposite polarization of theinterfering system will fix the problem. This is an easy and inexpensive solution to tryfirst, as it doesn’t require the repositioning of any equipment.

If that doesn’t work, try changing the frequency of your system. Systems on differentfrequencies tend not to interfere with each other. One simple way to change your

frequencies is to simply swap the transmit and receive frequencies on your system.

Troubleshooting Network Performance If you’re trying to pinpoint and troubleshoot problems in network performance, thefirst, best advice is to laboriously test and document your system, its configuration,maintenance, and anything else that you do to it. That way, should the network startoperating in a sub-par fashion, you have a history with which to compare it.

Change Management and Your Network There are two ways you can approach troubleshooting a network with performanceproblems. The first is to go in, oblivious to any changes and modifications that have been made to the system. That is, you go in to fix the problem, but have no clue whathas already been done. When this happens, the best you can do is start making changeshere and there, based on your educated guesses and experience, not on fact. The second,and obviously better, solution is to gather basic performance trend information and referto your network’s change management log so that you have a functional baseline fromwhich to begin the troubleshooting process.

A change management log is a document where you record each and every changeand bit of maintenance that is performed on your system, no matter how big, no matterhow small. If you installed a new router, that should be in the document, but so shouldsomeone going into the server room to reset a device.

NOTE There’s a story about a network technician performing the simple task of blowing dust out of a router’s fan. Ultimately, the dust was worked deeper into the fan, causing it to intermittently stop



660




and cause the router to overheat. Since the technician didn’t record this “simple” task in a change management document, it was never thought to be checked, until it was too late and the router

burned up.

It is also helpful, when making changes to your network’s configuration, to make asfew changes at once as possible. That way, if your network either takes a performancehit or goes down altogether, it’s easier to undo than if you’ve performed a dozendifferent things.

If you have a change management document and know when the system startedhaving problems, you can start analyzing changes that were made to the network andits devices. You might discover that a new routing protocol was introduced or a newQuality of Service policy was implemented. If you were to shoot blindly in the dark, itcould take you weeks to find these issues. If you have a change management document,however, it’s much easier to pin down the problem.

An effective, well-implemented change management plan has a number of usefulattributes that will help your overall network management and also aid in troubleshoot-ing. Benefits include the following:

▼ A checkpoint that allows you to measure performance, both before and after

changes are made to the network■ A journal of network updates, maintenance, and reconfigurations, allowing

you to compare your network and its changes to previous configurations inyour network’s history

▲ A rollback tool, which makes it easier to restore your system to an optimalconfiguration if the performance of a new configuration does not live up toyour expectations

For best results, you’ll have the proper software and hardware devices that will help

you gather and analyze your performance metrics. For some suggestions, flip back toChapter 13.

Router Performance Problems If you suspect there are performance problems with your router, consult your changemanagement document. Have you changed anything recently? Once a networkingdevice has been set up and is working, problems generally stem from a person tryingto improve the device’s performance. Assuming there isn’t some hardware problem (anunplugged cable or network card improperly seated), then the next place to look is ifthere were any changes made to the device’s configuration.

NOTE Don’t dismiss hardware problems too quickly. It is always possible that someone went to perform a seemingly unrelated task and accidentally pulled a power cord a bit too hard or pinched a network cable with a floor tile or the rack door. Always inspect your hardware before you commit hours of your time to sorting through configuration files. One of the biggest sources of network snafus



661Ch t 15 T bl h t i C i N t k




is cabling plugged into the wrong ports. Don’t be sheepish about preparing a map showing which cables go where between your devices and making sure that all your cables are labeled properly on both ends so that you can instantly find where the cable is plugged in. It beats the tedious alternative:pulling on cables to see where they go.

Hopefully, you backed up your router’s configuration file. Taking the few seconds to back up the file when it is working optimally will save you untold hours trying to restorethe system. The time to back up the configuration file is when everything is workingwell. Backups should also be made before and after every single change is committed tothe device. This allows you to see exactly what is different between the pre- and post-change configuration.

If you don’t have a backup of the configuration file, the next step is to study yourchange management documentation. This documentation should describe all thechanges that have been made to the router. Examine the document and see whichchanges might be responsible for your router’s problems. You might have to go back tothe router’s configuration file and undo those changes, one by one, until the problemhas been resolved.

NOTE Better yet, if you don’t have a backup of your router’s configuration file, put this book down right now and go make one. Don’t worry, we’ll wait for you.

The culprit might also be changes in the device’s operating system. If you’ve recentlyupgraded your operating system or applied a patch, that’s a good place to check. Beforeadding new operating systems or applying patches, you should understand just howyou can rollback the operating system to the previous, operational operating system ifsomething goes wrong. Remember, however you attack a troubleshooting problem, thegoal is to “follow the wire” and track the source of the problem down to the end.

Keep in mind that the most important thing to do when troubleshooting is to proceedcarefully and logically. Don’t change several variables at the same time. Make a change,observe (be patient), document if necessary, and then proceed to the next step. Fixingproblems in complex systems is more about process than luck. That said, we wish youthe best of luck in troubleshooting and in life.



