S O L U T I O N S

c o mm V e r eG • Hong Kong• Bangkok• Beijing • Brunei• Kuala Lumpur• Manila• San Jose• Seoul• Shanghai• Singapore• Taipei

Network Diagnostic Introduction

Kae Hsu

Commverge Solutions, Taiwan

FEGTS IP Training2011/11/11, Taipei

2

WW

W.C

OM

MV

ER

GE

.CO

M

Object

Course Object– By the course, the students could understand basic

network troubleshooting concept, principle and relative tools

Course Information– 2.5 hours lecture & hand-on practice– 30 minutes Q & A

3

WW

W.C

OM

MV

ER

GE

.CO

M

Agenda

Network diagnostic concept Hostname resolution verification Network connection verification Application condition verification Low level traffic investigation Friendly tools Hands-on practice

4

WW

W.C

OM

MV

ER

GE

.CO

M

Network diagnostic concept

Regular Internet connection concept

Client

DNS

Server

www.abc.com = ?

www.abc.com = 203.47.56.180

5

WW

W.C

OM

MV

ER

GE

.CO

M

Network diagnostic concept

Regular troubleshooting sequence– Hostname resolution verification

• nslookup & dig– Network connection verification

• ping & traceroute– Application condition verification

• telnet

6

WW

W.C

OM

MV

ER

GE

.CO

M

Hostname resolution verification

– nslookup• Name/IP address query

7

WW

W.C

OM

MV

ER

GE

.CO

M

Hostname resolution verification

– nslookup• Mail Exchange (MX) query

8

WW

W.C

OM

MV

ER

GE

.CO

M

Hostname resolution verification

– dig• Name/IP address query

9

WW

W.C

OM

MV

ER

GE

.CO

M

Hostname resolution verification

– dig• Mail Exchange (MX) query

10

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– ping• Check network connection status

– concept

Client Server

Are you there?

Are you there?

Are you there?

Are you there?

Are you there?

Are you there?

I am here

I am here

I am hereI am

here

I am here

I am here

I am here

“Server” is alive

11

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– ping• ICMP packet

– ICMP echo-request & echo-reply– Identify reachability & round-trip time

Client Server

echo request

echo request

echo request

echo request

echo request

echo request

echo reply

echo reply

echo replyecho

reply

echo reply

echo reply

echo reply

“Server” is alive

12

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– ping• ICMP identifier & sequence number

– match reply & request

Client Server

echo request

echo request

echo request

echo request

echo request

echo request

echo reply

echo reply

echo replyecho

reply

echo reply

echo reply

echo reply

13

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– ICMP block by network filter

Client Server

echo request

echo request

echo request

echo request

echo requestICMP timeout packet dropped

14

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– traceroute• Check packet forwarding path information

– concept (in forwarding path)• router will drop packet with TTL=1

– “ICMP time exceeded” message sent to source with router inbound interface

TTL=1

ICMP

TTL=2

ICMP

TTL=3

ICMP

TTL=1

TTL=2 TTL=1

15

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– concept (arrive destination)• destination will NOT check TTL status• different response with different probe packet

– ICMP echo-request – response ICMP echo-reply– UDP with high destination port – response ICMP port unreachable

• ICMP

• UDP

TTL=4

ICMPecho reply

TTL=3 TTL=2

TTL=4 TTL=3 TTL=2

TTL=1

TTL=1

ICMP port unreachable

16

WW

W.C

OM

MV

ER

GE

.CO

M

Network connection verification

– Multiple path in a single traceroute task• router load-share the traffic by flow information• identify different flow by

– different ICMP echo-request identifier– different UDP port number

TTL=3

ICMPTime Exceed

ICMPTime Exceed

ICMPTime Exceed

ICMPTime Exceed

17

WW

W.C

OM

MV

ER

GE

.CO

M

Application condition verification

Internet application communication concept– TCP 3 way handshaking

• Verify TCP connection first during troubleshooting

From "Figure 211: TCP “Three-Way Handshake” Connection Establishment Procedure" in TCP/IP Guide

18

WW

W.C

OM

MV

ER

GE

.CO

M

Application condition verification

– telnet• To verify the destination site service status

– example• A WEB service• check correct IP information• check network connection status• check service response

19

WW

W.C

OM

MV

ER

GE

.CO

M

Application condition verification

– example

20

WW

W.C

OM

MV

ER

GE

.CO

M

Low level traffic investigation

“Sniffer” the traffic– TCPDUMP

21

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

WinMTR– Probe target & provide path information together

• Download: http://winmtr.net/download-winmtr/

22

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

Looking glass– Execute ping/traceroute from different sites

23

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

Looking glass

24

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

Looking glass list

25

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

Wireshark

26

WW

W.C

OM

MV

ER

GE

.CO

M

Friendly Tools

– Wireshark reference guide• “Wireshark Network Analysis, The Official Wireshark Certified

Network Analyst Study Guide” by Laura Chappell

27

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

Lab environment

SSID: WL-330gEPWD: 0123456789192.168.1.0/24

J4350.3

C3750-1.1

C3750-2.2

192.168.7.6

192.168.1.X 192.168.2.X

192.168.3.X

192.168.4.X192.168.5.X192.168.6.X

192.168.7.X

Sniffer box

28

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

Install Wireshark– Download: http://www.wireshark.org/download.html

29

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

– Use Wireshark to monitor DNS message

30

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

– Use Wireshark to monitor ICMP message

31

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

ping

32

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

traceroute

33

WW

W.C

OM

MV

ER

GE

.CO

M

Hands-on practice

DNS – nslookup– 開始 ->

執行 -> “cmd”

34

WW

W.C

OM

MV

ER

GE

.CO

M

Prior Course Q & A Summary

Is there any troubleshooting skill for SCTP?– Using Tools

• Iperf over SCTP– Adapted version of Iperf(version 1.6.5), runs on lksctp– Use iperf with –z to open SCTP connection to test target

• Windows SCTP library– Bundle some SCTP application for simple test

– Useful link• http://www.sctp.be/

– SCTP research and simulation page– SCTP Software page– SCTP application Software production page

• http://sigtran.org– SCTP Test Tool (stt)– SCTP Performance Test

35

WW

W.C

OM

MV

ER

GE

.CO

M

Prior Course Q & A Summary

How to capture packet by tcpdump from TWO or more NICs at the same time?A. Use “any” as “-i” parameter on Linux

a) From tcpdump man page:-I

Listen on interface. If unspecified, tcpdump searches the system interface list for the lowest numbered, configured up interface (excluding loopback). Ties are broken by choosing the earliest match.On Linux systems with 2.2 or later kernels, an interface argument of ``any'' can be used to capture packets from all interfaces. Note that captures on the ``any'' device will not be done in promiscuous mode.If the -D flag is supported, an interface number as printed by that flag can be used as the interface argument.

36

WW

W.C

OM

MV

ER

GE

.CO

M

Prior Course Q & A Summary

How to flush DNS cache manually?– With BIND 9.2.0 or newer

• # rndc flush– With older BIND

• Kill BIND process and restart it• # rndc restart

– For detail information, please refer• “Flushing (Clearing) a Name Server's Cache”

from “DNS & Bind Cookbook” by Cricket Liu, O’Reilly

37

WW

W.C

OM

MV

ER

GE

.CO

M

Prior Course Q & A Summary

How to execute ping by different interface?– Windows platform

• Use ‘-S’ parameter to identify source IP address– Linux

• Use ‘-I interface/IP_address’ to identify source IP address– IOS

• Enter extended command to identify source IP or interface• Use “source” parameter to identify source IP address (newer)

– Junos• Use “source” parameter to identify source IP address

38

WW

W.C

OM

MV

ER

GE

.CO

M

Q & A