Cisco Security Troubleshooting: Part I – Connectivity Through …€¦ · · 2010-10-06Expert Reference Series of White Papers 1-800-COURSES Cisco Security Troubleshooting: Part

Expert Reference Series of White Papers

1-800-COURSES www.globalknowledge.com

Cisco Security Troubleshooting:

Part I – Connectivity Through ASA or PIX

Firewalls

Copyright ©2009 Global Knowledge Training LLC. All rights reserved. 2

Cisco Security Troubleshooting: Part I – Connectivity Through ASA or PIX FirewallsDouglas B. McKillip, P.E., CCSI, CCSP, CCIE #1851

IntroductionThis white paper is the first in a three-part series, each of which examines the challenge of implementing network security on equipment from Cisco Systems® while maintaining the connectivity requirements of the business or enterprise.

Cisco Security Troubleshooting

Part I: Connectivity Through ASA or PIX Firewall

Part II: Virtual Private Networks

Part III: Troubleshooting Intrusion Prevention Systems.

The focus in Part 1 will primarily be on the most effective use of both GUI-based and CLI-based troubleshooting tools. We will examine the following troubleshooting tools available on both the ASA and PIX firewalls except

• debug packet (PIX OS 6.3 only)

• capture

• packet tracer

Troubleshooting ScenarioThe following scenario will be used to illustrate the use of the troubleshooting tools outlined above. If the names and IP Addresses look familiar to some of you, this is because the diagram represents a portion of the lab topology used in Cisco® Security classes offered by Global Knowledge.


Figure 1

Debug PacketAs mentioned previously, the debug packet command is only available on PIX Firewalls, having been intro-duced as early as the 5.x generation of code. While “debug packet” is no longer supported beginning in version 7.0 of either the PIX or ASA code, we will later examine the use of the newer “real-time” option to the capture command as an alternative.

The following 2 @ debug packet commands were used to verify the TCP 3-way handshake originating from the Outside PC, 150.150.1.20 and the DMZ-Server, 172.16.1.15:

(initiated via http://200.200.1.15)

debug packet dmz src 150.150.1.20 dst 172.16.1.15 proto tcp dport 80

debug packet dmz src 172.16.1.15 dst 150.150.1.20 proto tcp sport 80

Note below that the attempt to verify both debugs only shows the first!

LABPIX(config)# show debugdebug packet dmz src 150.150.1.20 dst 172.16.1.15 proto tcp dport 80 both


Here is the live output (the raw hex and sequence #’s have been omitted).--------- PACKET ---------

-- IP --

150.150.1.20 ==> 172.16.1.15

ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x34

id = 0xf40e flags = 0x40 frag off=0x0

ttl = 0x7f proto=0x6 chksum = 0xc2eb

-- TCP --

source port = 0xa8f dest port = 0x50syn

--------- END OF PACKET ---------(output truncated)

--------- PACKET ---------

-- IP --

172.16.1.15 ==> 150.150.1.20

ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x2c

id = 0x501 flags = 0x40 frag off=0x0

ttl = 0x20 proto=0x6 chksum = 0x1102

-- TCP --

source port = 0x50 dest port = 0xa8fsyn ack

--------- END OF PACKET ---------(output truncated)

--------- PACKET ---------

-- IP --

150.150.1.20 ==> 172.16.1.15

ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x28

id = 0xf410 flags = 0x40 frag off=0x0

ttl = 0x7f proto=0x6 chksum = 0xc2f5

-- TCP --

source port = 0xa8f dest port = 0x50ack

--------- END OF PACKET ---------


Two points are worthy of note in the previous example. First, each debug packet command was used with as much qualification as possible to target the output to be relevant for troubleshooting the specific web connec-tion. Second, the use of the second simultaneous debug with the parameters carefully inverted results in the ability to see all of the http client-server exchange.

Capture – Part I – Output to TerminalThe capture command first became available in PIX OS 6.2; however, the real-time option only recently became available in release 8.0. In this next section we will examine the flexibility in how this valuable troubleshooting command can be used. Again, for all the alternative implementations examined here, the same scenario with the Outside-PC at 150.150.1.20 connecting to the DMZ-Server at 172.16.1.15 (http://200.200.1.15) will be used.

- Commands entered:

access-list WWW_TRAFFIC permit tcp host 150.150.1.20 host 172.16.1.15 eq 80

access-list WWW_TRAFFIC permit tcp host 172.16.1.15 eq 80 host 150.150.1.20

capture WWW-TEST-DMZ access-list WWW_TRAFFIC interface dmz real-time

- Terminal output:

(occurred immediately after entering the capture command above - actual HyperTerm line wraps are displayed)

Warning: using this option with a slow console connection may

result in an excessive amount of non-displayed packets

due to performance limitations.

Use ctrl-c to terminate real-time capture

1: 18:58:19.230258 150.150.1.20.3010 > 172.16.1.15.80: S 839932538:839932538(

0) win 64240 <mss 1260,nop,wscale 0,nop,nop,nop,nop>

2: 18:58:19.231891 172.16.1.15.80 > 150.150.1.20.3010: S 954995:954995(0) ack

839932539 win 8820 <mss 1460>

3: 18:58:19.232882 150.150.1.20.3010 > 172.16.1.15.80: . ack 954996 win 64240

4: 18:58:19.235400 150.150.1.20.3010 > 172.16.1.15.80: P 839932539:839932983(

444) ack 954996 win 64240

5: 18:58:19.262635 172.16.1.15.80 > 150.150.1.20.3010: P 954996:955068(72) ac


k 839932983 win 8376

6: 18:58:19.307753 150.150.1.20.3010 > 172.16.1.15.80: P 839932983:839933408(

425) ack 955068 win 64168

7: 18:58:19.314894 172.16.1.15.80 > 150.150.1.20.3010: P 955068:955372(304) a

ck 839933408 win 7951

8: 18:58:19.315382 172.16.1.15.80 > 150.150.1.20.3010: F 955372:955372(0) ack

839933408 win 7951

9: 18:58:19.316481 150.150.1.20.3010 > 172.16.1.15.80: . ack 955373 win 63864

10: 18:58:19.317015 150.150.1.20.3010 > 172.16.1.15.80: F 839933408:839933408(

0) ack 955373 win 63864

11: 18:58:19.317778 172.16.1.15.80 > 150.150.1.20.3010: . ack 839933409 win 79

51

11 packets shown.

0 packets not shown due to performance limitations.

Capture – Part II – Output to HTML FileAssuming the same access-list as shown above, if the real-time option is omitted the same text display as shown on the previous page can be derived by

show capture WWW-TEST-DMZ

A more readable format can be derived by securely viewing the capture buffer using a browser; however, before this can be done, the following commands must be entered into the PIX or ASA security appliance: (using the Admin-PC in our Lab Topology as an example)

http server enable

http 10.10.10.10 255.255.255.255 inside

To view the capture file in buffer memory, execute the following.

https://10.10.10.1/capture/WWW-TEST-DMZ

This will result in the following authentication prompt; the default credentials expected would be the enable password for the appliance (Password: field) with the User Name: blank


Screen 1

The result is shown below.

Screen 2

Note that not only is the output sorted into much more readable aligned columns, but the TCP flags are easily seen. (S - SYN, P - PSH, F - FIN)

Capture – Part III – Output to a Windows Pcap FileIf even more granular examination of the packets contained in the capture file is needed, the buffered capture file can be copied to a TFTP Server for later viewing using the following:

copy /pcap capture:WWW-TEST-DMZ tftp://10.10.10.10/WWW.cap

After confirming your selections in the TFTP CLI dialog, the resulting file can be viewed with the default protocol analyzer (“sniffer”) application associated with the .cap suffix. In this case, Wireshark® was used to generate the following display.


Screen 3

Capture – Part IV – Using the Packet Capture Wizard in ASDMIn this last sub-section on the capture command, we will examine how the Adaptive Security Device Manager (ASDM) for the ASA and PIX can be used to not only analyze packets in and out of a particular interface (as we have done previously), but also look at the overall flow through the security appliance as a whole. In the series of screens below, the same http packet stream is analyzed, except that the “outside” interface is a logical one with the name outside_lab and it is part of an 802.1q trunk. Note that in step 3 we need to specify the trans-lated address (200.200.1.15) of the DMZ-Server.

Screen 4


Screen 5

Screen 6

Note that in screen 6 above, bidirectional flows are shown for both the dmz and the outside_lab interfaces. Secondly, for the egress interface (outside_lab) the VLAN ID is seen, as the Ethernet frames entering and leav-ing use 802.1q trunk encapsulation. If the “Launch Network Sniffer Application” button is clicked, the following is seen as a pop-up: (note that the 802.1q frame decode is clearly seen in the middle window)


Screen 7

Packet-tracerThe packet-tracer command was first introduced into ASA and PIX OS 7.2 and can be executed both from the command-line and from within ASDM. The big advantage of this powerful and relatively new tool is that most simple, unencrypted (non-VPN) flows can be simulated without having to physically access either of the two connection endpoints. We will show in this final section of this whitepaper several uses of packet-tracer. First, for consistency, we will show the simulated flow of the Outside-PC connecting to the DMZ-Server with http. Later, we will show more complex translation scenarios for both inbound and outbound access.

Below is the simulated http://200.200.1.15 from the Outside-PC as shown in ASDM.


Screen 8

If each of the Phase items shown above are expanded by clicking on the “+” sign, what is obtained is virtually identical to performing this using the CLI command

packet-tracer input outside_lab tcp 150.150.1.20 1024 200.200.1.15 80

Note that the first phase item in the GUI display above titled “ACCESS-LIST” is missing from the output below; the CLI does not display the null MAC-address access-list as a phase.

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow


Phase: 2

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

static (dmz,outside_lab) 200.200.1.15 172.16.1.15 netmask 255.255.255.255

nat-control

match ip dmz host 172.16.1.15 outside_lab any

static translation to 200.200.1.15

translate_hits = 1, untranslate_hits = 3


NAT divert to egress interface dmz

Untranslate 200.200.1.15/0 to 172.16.1.15/0 using netmask 255.255.255.255

Phase: 3

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group OUTSIDE_IN in interface outside_lab

access-list OUTSIDE_IN extended permit tcp any host 200.200.1.15 eq www


Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:


Phase: 5

Subtype: np-inspect

Result: ALLOW

Config:

class-map inspection_default

match default-inspection-traffic

policy-map global_policy


class inspection_default

inspect http

service-policy global_policy global


Phase: 6

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:


nat-control





Phase: 7

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:


nat-control





Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:


Phase: 9

Type: FLOW-CREATION


Subtype:

Result: ALLOW

Config:


New flow created with id 96, packet dispatched to next module

Phase: 10

Type: ROUTE-LOOKUP

Subtype: output and adjacency

Result: ALLOW

Config:


found next-hop 172.16.1.15 using egress ifc dmz

adjacency Active

next-hop mac address 0080.c882.11dc hits 4

Result:

input-interface: outside_lab

input-status: up

input-line-status: up

output-interface: dmz

output-status: up

output-line-status: up

Action: allow

Several items are worthy of note in the display of the various “Phases” on the previous page.

1) UN-NAT (Phase 2) actually does the reverse address translation (global→local).

2) NAT (Phase 6) is checking for Reverse Path Forwarding, or the spoofing of a source address associated with a different interface other than the one on which it arrived. While this could be enabled, it has not been here. (default = disabled)

3) NAT (Phase 7) looks at the exceeding of specified maximum or embryonic connection limits, neither of which are specified in this example.

The following ASDM screenshot shows the power of the integrated use of this tool with the GUI display of rel-evant access lists; it was generated by clicking on the Show rule in Access Rules Table link: (note that the relevant access-list entry is highlighted.)


Screen 9

Our final example for the use of packet-tracer will be for a complex set of Network Address Translation (NAT) rules as might be encountered for an organization requiring VPN connectivity as well as general clear-text con-nectivity to the Internet. To visualize this scenario, the following shows a revision to the troubleshooting scenario presented previously – namely, the addition of a remote location to which a site-to-site VPN session might be initiated. To accommodate this, what is frequently needed is a NAT exemption rule, one that won’t translate the 10.10.10.0/24 network-sourced packets traversing the VPN tunnel to 10.20.10.10/24. The resulting output shows what happens when provision was not made for that exemption.


.2

.1

.1

INTERNET

DMZ-Srv

172.16.1.15

NAT:

200.200.1.15

Outside-PC 150.150.1.20

.1

Perimeter

Router

Admin-PC (Syslog Server)

10.10.10.10

ASA/PIX

DMZ Subnet

172.16.1.0/24

200.200.1.0/24

XP

10.10.10.0/24

10.20.10.0/24

Figure 2

Below is the result for the command:

packet-tracer input inside icmp 10.10.10.10 0 0 10.20.10.10Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:


Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:



in 0.0.0.0 0.0.0.0 outside

Phase: 3

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:


Phase: 4

Type: INSPECT

Subtype: np-inspect

Result: ALLOW


Result:

input-interface: inside

output-status: up

input-status: up


input-line-status: up

output-interface: outside

Phase: 5

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

static (inside,outside) udp interface syslog 10.10.10.10 syslog netmask 255.255.255.255

nat-control

match udp inside host 10.10.10.10 eq 514 outside any

static translation to 200.200.1.1/514



Phase: 6

Type: NAT

Subtype:

Result: DROP


Config:

nat (inside) 1 0.0.0.0 0.0.0.0

nat-control

match ip inside any outside any

dynamic translation to pool 1 (200.200.1.2 [Interface PAT])



output-status: up


Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

As can be seen above, instead of a NAT exemption rule being matched, the 10.10.10.10 IP Address would be translated to the ASA outside interface via Port Address Translation (PAT). As a result, if users attempted to ping across the secure tunnel, their attempts would fail. Further VPN troubleshooting techniques will be addressed in the second whitepaper in this series, Part II: Virtual Private Networks. Part III of this series examines Trouble-shooting Intrusion Prevention Systems.

SummaryOne key concept common to all the preceding scenarios is an effort to “target” the troubleshooting as much as possible. This minimizes the extraneous output, which usually has no relevance to the problem at hand. An added bonus is minimizing the impact on the CPU of the appliance that would be understandably burdened by a more generalized “debug any” approach and could negatively impact throughput. We have seen the effective use of the debug packet tool on older operating system, and newer capture and packet tracer tools, the last of which can be used without involving the end-user.

Learn MoreLearn more about how you can improve productivity, enhance efficiency, and sharpen your competitive edge. Check out the following Global Knowledge courses:

IINS (Introduction to IOS Network Security)

SNRS (Securing Networks with Cisco® Routers and Switches)

SNAF (Securing Networks with ASA Fundamentals)

SNAA (Securing Networks with ASA Advanced)

MARS (Cisco® Monitoring Analysis and Reporting System)

CANAC (Cisco® Appliance for Network Admission Control)

For more information or to register, visit www.globalknowledge.com or call 1-800-COURSES to speak with a sales representative. Our courses and enhanced, hands-on labs offer practical skills and tips that you can imme-diately put to use. Our expert instructors draw upon their experiences to help you understand key concepts and


how to apply them to your specific work situation. Choose from our more than 700 courses, delivered through Classrooms, e-Learning, and On-site sessions, to meet your IT and management training needs.

About the AuthorDouglas B. McKillip, P.E., CCIE #1851 is the president and principal consultant of Innovative Integrators Incorpo-rated, a Delaware Corporation actually based in Delaware.

In addition to a BS and MS in Chemical Engineering from M.I.T., Doug also later obtained an M.S. from the University of Delaware in Computer and Information Science. After 15 years of experience at DuPont and a brief stint with the original startup company associated with the Raptor EagleTM Firewall, Doug began his now 15+ year career of teaching and consulting, specializing in Internet Security with hardware from Cisco Systems® since 1998.

Doug’s hobbies include playing piano at his local church and physical activities such as running, cycling, swim-ming, golf, skiing, and windsurfing. He and his wife Karen reside in Wilmington, Delaware. Doug can be reached at [email protected].

Documents

Cisco Security Troubleshooting: Part I – Connectivity Through …€¦ · · 2010-10-06Expert Reference Series of White Papers 1-800-COURSES Cisco Security Troubleshooting: Part