July 19th, 2004 Joint Techs, Columbus, OH 1

Monitoring the10 Gigabit Abilene Backbone

Jörg Micheel<joerg@nlanr.net>

July 19th, 2004 Joint Techs, Columbus, OH 2

Passive measurement hardware

• Collect all data without loss (buffer, indicate loss)

• Flexible payload discard, early• Provide accurate local/global timing of

packet arrivals• Have host CPU(s) available for analysis

and display• Optionally accelerate analysis computation

July 19th, 2004 Joint Techs, Columbus, OH 3

PCI bus vs. network linksBus MByte/sec

PCI 32@33 132

PCI 64@33 264

PCI 64@66 528

PCI-X @ 66 528

PCI-X @ 100 800

PCI-X @133 1064

Network link MByte/sec

OC3c RX 20

OC12c RX 75

OC48c RX 311

OC192c RX 1244

Gigabit 125

10 Gigabit 1250

July 19th, 2004 Joint Techs, Columbus, OH 4

NICs vs. NMCs• NICs have inherent packet rate limit due to per

packet buffer and interrupt handshake• Interrupts are more costly than ever before!• Do away with classic NIC driver, specialized

driver and API library• NMCs use single circular buffer with overflow

control• Handshake between HW and application• Zero copy data transfer (DMA into user space)• Network analysis at uncached RAM

performance (i.e. 1450 MByte/sec on Dell 2650)

July 19th, 2004 Joint Techs, Columbus, OH 5

Some performance tests• GigE cards: D-Link, SysKonnect 9843,

DAG4.3GE• OC48c PoS cards: Lucent OptiStar, Dag4.3

OC48c• libpcap interface (straight, mmap), DAG API

library• Fake application: CRC32 computation• Result: line rate performance for all NMCs• Result: can do full bidirectional Gigabit to RAID0

array

July 19th, 2004 Joint Techs, Columbus, OH 6

DAG6 OC192c/10GigE

July 19th, 2004 Joint Techs, Columbus, OH 7

Abilene backbone 2004

July 19th, 2004 Joint Techs, Columbus, OH 8

IPLS installed NLANR gear

• Two NLANR OC192MONs, one OC48MON, CDMA time synchronization via Praecis Ct and TDS-24

• Instrumented all present Abilene backbone links (IPLS-KSCY, IPLS-CHIN, IPLS-ATLA)

• Planned to stay there for the lifetime of the backbone

July 19th, 2004 Joint Techs, Columbus, OH 9

Measurement and Analysis• Collect substantial backbone data set (Abilene-3)• Stratified random sampling over weeks (Abilene-4)• Landspeed record load; impact studies on application

and network performance• Monitoring TCP happiness via real time application• NLANR real time applications• Router delay studies• More challenges from collaborators!

July 19th, 2004 Joint Techs, Columbus, OH 10

Abilene-3 data set analysis

July 19th, 2004 Joint Techs, Columbus, OH 11

Acknowledgements

• Koryn Grant, Endace• Matt Zekauskas, Rick Summerhill,

Caroline Carver, Internet2• John Hicks, TransPAC/Indiana

July 19th, 2004 Joint Techs, Columbus, OH 12

References

• http://198.202.123.33/Special/ipls3.html• http://198.202.123.33/Sites/ipls-2004/• http://198.202.123.33/Special/ipls3/• ftp://198.202.123.33/traces/long/ipls/3/