APII/APAN Measurement Framework:Advanced Network Observatory

Masaki Hirabaru (NICT),Takatoshi Ikeda (KDDI 　 Lab),Motohiro Ishii (QIC), and Yasuichi Kitamura (NICT)

January 21, 2006APII/APAN Network Engineering

3 Case Studies

Acknowledgement to JaeHwa Lee @ KOREN

KwangjuBusan

Fukuoka

Korea

                                       

2.5G SONET

KORENTaegu

Daejon

Seoul XP

Genkai XP Fukuoka Japan

1,000km

APII/JGNII

Case 1: APII Seoul-Tokyo Redundant Connection

Tokyo XP

JGNII

TransPAC2

Southern Route(10G)

Northern Route(1G)

10G

10G

JGN II : Frequent Maintenance Outage but No Fault Detection and Redundancy!

Establish Backup L2 VLAN Path (northern route)Switchover-Time Measurement on APII link was

37 secsimproved by decreasing OSPF Hello Interval (40 to 1),

5 secsNext: Seoul-Tokyo E2E Path

Needs a measurement point and NOC support in Korea

2005-05-16Tokyo – Fukuoka Northern Route

Tokyo – Fukuoka Southern Route

switc

ho

ver

time

Case 2: APII Route Propagation DelaysBGP Announce Delay

0

25

50

75

100

Koganei Tokyo Fukuoka Busan Daejon Seoul

Sec

s

Minimum Average Maximum

BGP Withdraw Delay

0

25

50

75

100

Koganei Tokyo Fukuoka Busan Daejon Seoul

Sec

s

Minimum Average Maximum

busan

daejon

seoul

tokyo4

tokyo5

tokyo

Generator at Koganei (A / W) 2-hour intervals

monitor

AS7660

AS64582

133.69.36.0/24http://mrtg.koganei.itrc.net/beacon/

AS20965

AS2907

AS6509

AS2523

JGNII

TransPAC2

AS9270

fukuoka

AS10746

koganei

AS2020

Data collected on January 19, 2006

Case 3: Common Measurement Machines and Traffic Monitoring for APII

Fukuoka – Busan TCP Performance MeasurementThe statistics were last updated Monday, 26 September 2005 at 19:45

Max  =>: 803.0 Mbps Average  =>: 791.0 Mbps Current  =>: 799.9 Mbps Max  <=: 923.0 Mbps Average  <=: 883.2 Mbps Current  <=: 875.8 Mbps

*Theoretical wire rate of TCP on GbE is 941 Mbps at 1500B MTU.No wire rate confident box has not been available in KOREN at that time.

Busan BGP Instability Measurement

Weathermap

SmokePing with KOREN NOC from Tokyo XP

APAN/APII Measurement Framework for NOCs and Researchers:Advanced Network Observatory

Two Targets Reliable Operations

Research Support

NOC Daily OperationsSupported by Stable Measurement

NOC BNOC C

NOC A

Research Activityof Advanced Measurement Provide data

and environment

Other Observatories Common data export

Observatory at Tokyo XP (1)NOC DATASET,TOOL and MACHINES

Data set description tool machine

Throughput 1G throughput Iperf & BWCTL nms1

10G throughput Iperf & BWCTL nms5

One-way delay one-way delay OWAMP nms4

netflow Flow data flow-tools nms3

Usage statistics

Interface usage ucd-snmp & SNAPP nms2

Core router A(tpr4)

Core router B(tpr5)

nms1

nms2

nms3

nms4 L2 S

W

nms5

10G10G

1G

NETWORK DIAGRAM

perf4 perf5 perf6 bwctl

L2 SW

NOC machinesResearch machines supported by NOC

NOC machine

Observatory at Tokyo XP (2)SPECIFICATION of MACHINES

machine OS CPU Memory Disk NIC cost

nms1,2 Linux

2.4.6

+ web100

Dual Xeon

3.6GHz

DDR266

1GB x 2

(nms1,2,4)

U320 SCSI 72G

(nms3)

U320 SCSI 144G

2 x 1000base-TX

(Onboard)

(only nms1)

1 x1000base-SX

( Intel NIC)

around

$3,000

nms3,4 FreeBSD

5.2.1

nms5 Linux

2.6.6

Dual Opteron

2.2GHz

PC3200

1GB x 2

IDE 2 x 1000base-TX

(Onboard)

1 x 10G-LR

(Chelsio

T210-LR)

around

$8,000

(PC $4k,

NIC $4k)

-10G NICs vendors: Intel (earliest), Neterion (s2io, PCI express, PCI-X version 2), Chelsio (w/ or w/o TOE), Mirinet (PCI express)

- Remote console access

PCI-X 64bit/133Mhz

PCI-X 64bit/133Mhz

Example Configuration for G-H 10G Link- Minimum Requirements for Operations -

TARGET DATA– Throughput (10G)– One-way delay

EQUIPMENTS (each node)– machineA (for throughput)

same spec of nms5

– machineB (for One-way delay)same spec of nms1-4

– Router or Switch port 10G x 1 (for machineA) 10/100/1000 x 2 (for machineA,B)

COST (each node)– Servers machineA $8,000 machineB $3,000

– Router or Switch port depend on vender

– Rack space for research equipment free?

Fukuoka

Korea

KOREN

10G

Genkai XP

Kitakyushu

Fukuoka Japan

APII

Tokyo XP

10G

Tokyo Japan

Current Genkai XP Configuration

Tokyo XPsummit400summit5it

Genkai XP (Fukuoka)

APII Research Servers

JuniperM10

JGN II

summit1it

TAP

FI 4802Genkai-perf1

Genkai-perf2

One-way delayservers(Zhang)

Genka-bwctl

Busan

10G

Genkai-ia64

10G

Research 10G measurement is ready Genkai XP (QIC’s) operational NOC support Ready for collaboration under Observatory framework A local 10G port for measurement is needed to isolate a problem from JGN II Fukuoka – Tokyo link.