4-CEN Fault Isolation

CEN Fault Isolation

2

1. Checking Services using OTNM2000 GUI

2. Checking Services using Command Line

Interface (CLI)

3. Fault Isolation

Contents

3



Interface (CLI)

3. Fault Isolation

Contents

4

Checking Services using OTNM2000 GUI

1. The Task Panel

ALARM: To view the current alarms

ALARM_HIS: To view the historical alarms

PERFORMANCE: To view NE current performance (e.g.

IOP(Input Optical Power/Received) and OOP(Output Optical

Power/Transmit), Rx/Tx , NE/FAN Temperature, Laser Bias,

Voltage, Memory Utilization

PERFORMANCE_HIS: Querying Performance Code

STATUS: To view the current port Link State, Optical Module

Information, Received and Transmitted Packets.

Card Configuration: FAN Speed Configuration, Clock Config.

MASK SETTING: Alarm and Performance masking/hiding

Control Command: Loopback Testing, Card Soft Reset, Card

Switching to Main/Protection, Port Laser ON/Off,

5


2. The Help Toolbar

Alarm and Performance Help: Views the relevant information of the alarm

and performance.

6


ALARM

To view the current and

timestamp of alarm: Right

click at the NE and select

“Open”. Click the alarmed

card.

Click the ALARM icon found

on the Task Panel.

7


ALARM_HIS

Users can know about the faults of

network via viewing alarm history, so

as to facilitate the fault analysis.

8


PERFORMANCE Users can know whether the equipment or service in the network is abnormal or not

by viewing performances. Viewing performance also help the maintainers to handle

the faults in a timely manner and prevent faults.

9


PERFORMANCE_HIS View the performance history to obtain the abnormal performance data, in order to

instruct the current routine operations.

10


STATUS

Via viewing card status, users can

check whether the cards are

present and the software /

hardware version complies with

the planning.

11


Card Configuration

Via card configuration, users can enable E1 ports, configure clock and time

synchronization.

12


Control Command

Via control command, users can perform software reset, card forced switch, E1

loop testing and laser on/off of optical ports.

13



Interface (CLI)

3. Fault Isolation

Contents

14

Checking Services of Dynamic Networking

Check IS-IS, OSPF, and BGP neighbor establishment.

Check RSVP and LDP session setup.

Check whether BGP distributes direct routes.

Check VC establishment.

Check the BFD status.

Check the MPLS and VRF ILM forwarding tables.

Check services through LSP ping, VRF ping, and Internet Control

Message Protocol (ICMP) ping.

15

OSPF check command:

router#show ip os ne

Output:

OSPF process 100:

Neighbor ID Pri State Dead Time Address Interface Instance

ID Up Time

6.6.6.6 1 Full/ - 00:00:30 56.1.1.2 XGE0/6/1.31 0

01d00h00m

22.22.22.22 1 Full/DR 00:00:34 115.1.1.2 GE0/11/3.31 0

01d00h31m

Service Self-test

Perform service self-test after service configuration.

16

IS-IS check command:

router#show clns is-neighbors

Output:

Area 300:

System Id Interface State Type Priority Circuit Id Up

Time

0101.1213.6185 XGE0/3/1 Up L1 64 0101.1213.6185.03

83d05h51

0101.1213.6008 XGE0/4/1 Up L1 64 0101.1213.6186.03

12d19h58

Service Self-test


17

Service Self-test

Check RSVP and LDP session setup.

Command:

router#show rsvp session

Output:

Ingress RSVP:

To From State Pri Rt Style Labelin Labelout LSPName DSType UpTime

1.1.1.1 5.5.5.5 Up Yes 1 1 SE - 131587 1-5.1-1.1.1e00000f ELSP_SIG 00:01:32

1.1.1.1 5.5.5.5 Up No 1 1 SE - 131584 1-5.1-1.1.1e00000f ELSP_SIG 00:02:26

Total 2 displayed, Up 2, Down 0.

Egress RSVP:

To From State Pri Rt Style Labelin Labelout LSPName DSType UpTime

5.5.5.5 1.1.1.1 Up Yes 1 1 SE 131589 - 1-5.1-1.1.1e00000f ELSP_SIG 00:01:30

5.5.5.5 1.1.1.1 Up Yes 1 1 SE 131585 - 1-5.1-1.1.1e00000f ELSP_SIG 00:02:17

Command:

router#show ldp session

Output :

Peer IP Address IF Name My Role State KeepAlive Up Time

8.8.8.8 GE0/11/3.31Passive OPERATIONAL 60 01d00h44m

6.6.6.6 XGE0/6/1.31Passive OPERATIONAL 60 01d00h12m

22.22.22.22 GE0/11/3.31Passive OPERATIONAL 60 01d00h44m

18

Service Self-test

Check VC establishment.

Command:

router#show mpls vc-table

Output:

VC-Name VC-ID Access-Intf Network-Intf In Label Out Label Tunnel-Label Nexthop

Status Oam-Status Up-Time Y1731

1-8.1-5.1 1 GE0/7/3.32 GE0/7/1.31 8713 132364 9219 5.5.5.5

Inactive NULL 00:49:30 0

1-8.1-6.2 2 GE0/7/3.32 GE0/7/2.31 8715 132096 9223 6.6.6.6

Active Bfd-UP 00:49:30 0

1-8.1-5.3 3 GE0/7/3.33 GE0/7/1.31 8714 132365 9216 5.5.5.5

Active Bfd-UP 00:49:30 0

1-8.1-6.4 4 GE0/7/3.33 GE0/7/2.31 8718 132097 9224 6.6.6.6

Standby Bfd-UP 01d20h52m 0


19

Service Self-test

Check the BFD status.

Command:

router#show bfd session

Output:

Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr

1801 1806 IPv4 Single-Hop Up 1d01h00m 45.1.1.1/32

1803 802 IPv4 Single-Hop Up 1d01h00m 115.1.1.2/32

1 1 MPLS LSP Single-Hop Up 1d00h30m 127.0.0.12/32

1804 0 IPv4 Single-Hop Down 00:00:00 56.1.1.2/32



7 6 MPLS VCCV Single-Hop Up 00:53:30 127.0.0.12/32


20

Service Self-test

Check the MPLS and forwarding tables.

Command:

router#show mpls forwarding-table

Output:

Codes: > - selected FTN, p - stale FTN, B - BGP FTN, K - CLI FTN,

L - LDP FTN, R - RSVP-TE FTN, S - SNMP FTN, I - IGP-Shortcut,

U - unknown FTN

Code FEC Tunnel-id FTN-ID Pri Nexthop Out-Label Out-Intf LSP-Type LSP-

ID Ingress-IDUp-Time

R> 1.1.1.1/32 1 1 Yes 45.1.1.1 131587 XGE0/6/4.31 1 5.5.5.5

ELSP_SIGNAL 00:25:54

R 1.1.1.1/32 1 20 No 56.1.1.2 131592 XGE0/6/1.31 2 5.5.5.5


R> 1.1.1.1/32 5 17 Yes 45.1.1.1 131588 XGE0/6/4.31 1 5.5.5.5


R> 2.2.2.2/32 2 19 Yes 56.1.1.2 131593 XGE0/6/1.31 1 5.5.5.5

ELSP_SIGNAL 01d00h31m

L> 6.6.6.6/32 0 2 Yes 56.1.1.2 132102 XGE0/6/1.31 LSP_DEFAULT

01d00h20m


21

Service Self-test

Command: FEC to Next Hop Label Forwarding Entry Table

router#show mpls ftn-table

Output:

Primary FTN entry with FEC: 1.1.1.1/32, id: 1, row status: Active

Owner: RSVP, Action-type: Redirect to Tunnel, Exp-bits: 0x0, Incoming DSCP: none

Tunnel id: 1, Protected LSP id: 1, QoS Resource id: 71, Description: 1-5.1-1.1.1e00000f

Select: 0

, Bfd id:2, MPLS-TP OAM Y1731 Initiative id:0, Lock:0 Cross connect ix: 290, in intf: - in

label: 0 out-segment ix: 269

Owner: RSVP, Persistent: No, Admin Status: Up, Oper Status: Up

Out-segment with ix: 269, owner: RSVP, out intf: XGE0/6/4.31, out label: 131587

Nexthop addr: 45.1.1.1 cross connect ix: 290, op code: Push

LSP Type: ELSP_SIGNAL

Class_Exp mapping:

CLASS DSCP_value EXP_value

be 000000 0

Bidirectional Forwarding Detection is configured

Bidirectional Forwarding Detection is enabled

Up Time: 00:26:05


22

Service Self-test

Command: MPLS Incoming Label Map

router#show mpls ilm-table

Output:

Codes: > - selected ILM, p - stale ILM, K - CLI ILM

Code In-Label Out-Label In-Intf Out-Intf Nexthop FEC ILM-ID

LSP-Type Up-Time

> 132370 8197 XGE0/6/1.31 GE0/11/3.31 115.1.1.2 22.22.22.22/32

579 LSP_DEFAULT 01d00h36m

> 132262 N/A XGE0/6/1.31 inloopback 0.0.0.0 192.168.184.0/24

747 LSP_DEFAULT 01d00h36m

> 132166 N/A XGE0/6/1.31 inloopback 0.0.0.0 192.168.88.0/24

651 LSP_DEFAULT


23

Service Configuration

Before using the show running command to query single-station configurations,

log in the device through the SecureCRT software.

SecureCRT-based login:

Both the username and password are root.

IP address of the

loopback interface

of the device

24


The following page is displayed after successful login to the core switch card.

25


Enter the following command and press Enter to log in to the control program.

Log in to the control

program.

26


Enter the password (a) and run en, as shown in the following figure.

27


The following describes the outputs of the show running-config command at

stations A1 and B1.

!

no service password-encryption

!

hostname A1_YZXTD

!

password a

log file rcu.log

! mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X 101 222.173.0.168 control-word

mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X 102 222.173.0.167 control-

word

mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO 201 222.173.0.168 control-word

mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO 202 222.173.0.167 control-

word

Username

Password

Active VC for 1X services

Standby VC for 1X services

A1

28


mpls class-type ct0 default

mpls class-type ct1 A11

mpls class-type ct3 a3





mpls te-class te0 default 0

mpls te-class te1 A11 1

mpls te-class te2 A11 2

mpls te-class te3 a3 3




Configure BFD FOR PW

mpls te-class te7 default 7

A1

29


mpls bfd l2-circuit 101 local-discriminator 101 remote-discriminator 101 min-tx 50 min-

rx 50


rx 50


rx 50


rx 50

mpls propagate-ttl

mpls vpws YZXTD_A11-LXRJ_B1-vpws-1X 101

vpws-vc YZXTD_A11-LXRJ_B1-pw-main-1X

vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-1X

mpls vpws YZXTD_A11-LXRJ_B1-vpws-DO 201

vpws-vc YZXTD_A11-LXRJ_B1-pw-main-DO

vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-DO

Configured BFD for PW.

Configured the VPWS.

Bound the VC to the VPWS.

A1

30

Service Configuration router ldp

router-id 3.76.1.201

explicit-null

no-php

global-merge-capability merge-capable

control-mode ordered

label-retention-mode liberal

pw-status-tlv

targeted-peer ipv4 222.173.0.168

exit-targeted-peer-mode

targeted-peer ipv4 222.173.0.167

exit-targeted-peer-mode

transport-address ipv4 3.76.1.201

hold-time 15

hello-interval 5

targeted-peer-hold-time 60

targeted-peer-hello-interval 20

keepalive-timeout 60 keepalive-interval 20

targeted-peer-hello-receipt

advertisement-mode downstream-unsolicited

mpls ldp lsp trigger host-only

Configured LDP. The IP address of the loopback

interface is configured here.

Configured the IP address for

loopback in the local station.

Quit the TAR-PEER configuration mode.

Configured the IP address of the loopback

interface for oppostation end B1.

all: LDP advertises labels for all.

host-only: LDP advertises label for hosts only.

for vc: LDP advertises labels for VCs only.

A1

31


!

interface FE0/5/5

mtu-board 9600

no shutdown

switchport

physical basic oe-mode electric

commit

! interface FE0/5/5.33

description for QoS-NMS

no shutdown

switchport

uni stream-type 1

uni stream-id 1 ploy 0 assign 1

uni stream-id 1 ovlan 33

mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X ethernet

mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X ethernet secondary

vc-mode revertive

commit

Configured the physical

interface.

Configured the subinterface of VLAN

33 in 1X services.

Enabled the interface.

Configured the interface

on the UNI side.

Bound the VC to this interface.

A1

32

Service Configuration Station A1:

!

interface FE0/5/6

mtu-board 9600

no shutdown

switchport

physical basic oe-mode electric

commit

!

interface FE0/5/6.34

description for QoS-other

no shutdown

switchport

uni stream-type 1

uni stream-id 1 ovlan 34

mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO ethernet

mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO ethernet secondary

vc-mode revertive

commit

33

Service Configuration !

interface GE0/5/1

mtu-board 9600

no shutdown

physical advance out-limit no-shutdown

physical queue-dispatch queue-no 0 mode 1 weight 1

physical queue-dispatch queue-no 0 cir 0 pir 1000 cbs 256 pbs 256













label-switching

commit

Enable label switching.

A1

34


!

interface GE0/5/1.31

description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)

ip address 10.1.1.5/30

no shutdown

label-switching

commit

ip ospf network point-to-point

ip ospf cost 100

enable-ldp ipv4 no-igp-sync

Configure dthe IP address of

the subinterface.

Enable OSPF on this

interface.

Enable LDP on the NNI. A1

35

Service Configuration !

interface GE0/2/1

mtu-board 9600

no shutdown

physical advance out-limit no-shutdown















label-switching

commit

A1

36


!

interface GE0/2/1.31

description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)

ip address 10.1.1.1/30

no shutdown

label-switching

commit

ip ospf network point-to-point

ip ospf cost 100

enable-ldp ipv4 no-igp-sync

!

A1

37


!

router ospf 2

ospf router-id 3.76.1.201

timers spf exp 50 50000

area 0.0.0.1 stub

network 3.76.1.201/32 area 0.0.0.1

network 10.1.1.0/30 area 0.0.0.1

network 10.1.1.4/30 area 0.0.0.1

cspf disable-better-protection

!

telnet max-clients 10

line vty

exec-timeout 0 0

login

!

dhcp enable

dhcp hops threshold 16

!

end

Configured OSPF. Set the IP address of the OSPF

interface to the IP address of the

loopback interface.

Added the loopback

interface to area 1.

Added the link interface to area 1.

A1

38

CLI basic commands in total

39

CLI : show running

Screenshot below: (CLI in the red box)

View global configuration

40

CLI : show arp

Screenshot below: (CLI and Content displayed as red font)

View all ARP

41

CLI : show ip interface brief


View port information

42

CLI : show ip route


View the IGP route table

43

CLI : show rsvp session


View RSVP session information

44

CLI : show rsvp interface


View RSVP enabled interface information

45

CLI : show mpls forwarding-table


View all LSP initiated by this router

46

CLI : show mpls ilm-table


View all mpls ilm table information

47

CLI : show bfd session


View bfd session information

48

CLI : show bfd interface


View bfd enabled interface

49

CLI : show ldp session


View LDP session information

50

CLI : show ldp interface


View LDP enabled interface information

51

CLI : show mpls vc-table


View all VC FEC statistics

52



Interface (CLI)

3. Fault Isolation based on different scenario

Contents

53

Fault Isolation

E-CES(E1) Service

Situation : Field Operations is conducting loop testing end-to-end(RNC to end site) but they

cant see the loop provided or the data cant pass thru RNC from the Source Site.

Checking:

1. Check the slot/port of the Source Site and Destination Site if mapping is created on

Business Management View.

2. Check the NE E1K5/E101/S101/S102 Card for current alarms. If seen PPI_LOS alarm,

physical termination fault. If seen AIS alarm, possible that alarm is on far end site.

3. If no alarm seen, check the port if unshut. Go to Secure CRT, type show run int <port>

command or go to STATUS>Link State must be with “on” status.

4. If port is up, check the status of VC by running the command show mpls vc-table and

check its uptime status. If with INACTIVE status, check the status of the tunnel by running

the command show rsvp session.

5. If no result show upon running the mpls vc-table command, check the LDP session if

configured by running show ldp session.

6. If RSVP session is down, check for active fiber cut.

6. If all protocols are up, conduct loop testing with the Engineer on site.

54


Control Command

E1 Loop Testing: Verify the configured E1 ports on Business Management .

Accessing C31 Ilagan on FH NMS:

LLB: Local Loopback = enable loop at Site A DDF

facing the BER tester(also installed in Site A)

DLB: Device Loopback = enable loop at Site A

facing the DDF of Site B(depending on the created

tunnel/trail, where BER tester is installed in Site B).

SITE A SITE B

55

Fault Isolation

E-LAN(Data Services (3G, BRO, Wimax, FD/TD-LTE)

Down Service

Checking:

1. Check the slot/port of the Source Site and Destination Site if mapping is

created on Business Management View.

2. Check the port for current alarms. If seen RLOS alarm, possible physical cable

fault. If seen LINK_LOS alarm, possible port setting problem(duplex/speed).

3. If no alarm seen, check the port if unshut. For R865, select the card, view the

STATUS>>Port Status to check for link status, received/transmit packets, current

speed/duplex mode.

4. If port is up but no packets, check the status of VC by running the command

show mpls vc-table and check its uptime status. If with INACTIVE status, check

the status of the tunnel by running the command show rsvp session.

5. If no result show upon running the mpls vc-table command, check the LDP

session if configured by running show ldp session.

6. If RSVP session is down, check for active fiber cut.

7. If no fiber cut, check for broken tunnel by running show run rsvp-path.

56

Fault Isolation

L2 services are unavailable.

1) Run show mpls vc-table to check whether the VC is normal.

Vc-name: VC name Vc-id: VC ID Access-intf: bound interface

Network-intf: network interface In label: incoming label Out label: outgoing label

Tunnel-label: label of the bound tunnel Status: status

Nexthop: IP address of the next-hop interface Up-time: establishment time

57

Fault Isolation

The VC is abnormal.

58

Fault Isolation

2) If the VC is abnormal, run show mpls forwarding-table to check the LSP

labels generated by LDP. The outgoing label is normal.

59

Fault Isolation

The outgoing label is abnormal.

60

Fault Isolation Run show mpls ilm-table to check the incoming label. The incoming label is normal.

61

Fault Isolation

The incoming label is abnormal.

62

Fault Isolation

3) If the LDP connection is abnormal, run show ip ospf neighbor to check OSPF.

Neighbor id: IP address of the peer loopback interface Pri: priority State: status

Dead time: The connection is interrupted if no Hello packet is received from the peer end within 30

seconds.

Address: IP address of the peer interface Interface: local interface

Instance id: local OSPF process ID (0 by default) Up time: establishment time

63

Fault Isolation The OSPF connection is abnormal.

64

Fault Isolation

If the OSPF connection is abnormal, ping the directly-connected interface. If the

directly-connected interface can be successfully pinged, OSPF is faulty.

65

Fault Isolation The directly-connected interface cannot be pinged.

5) If the directly-connected interface cannot be pinged, check the link state to

see whether the optical fiber is correctly connected.

66

THANK YOU

Documents

4-CEN Fault Isolation