The International Grid Testbed: a 10 Gigabit Ethernet success story

in memoriam Bob Dobinsonin memoriam Bob Dobinson

GNEW 2004, Geneva

Catalin Meirosuon behalf of the IGT collaboration

GNEW 2004 2

International

Testbed Contents

• www: who, why, what is IGT

• Native Ethernet over lightpaths: from 1 Gbps to 10 Gbps

• First demonstration of transatlantic native 10 GE over legacy OC-192 infrastructure

• Future plans

• Conclusions

GNEW 2004 3

International

Testbed Who’s who in IGT• Universities and Research Institutes

– Canada• Carleton University, Ottawa• University of Alberta• University of Victoria• TRIUMF, Vancouver• Université de Montreal• McGill University, Montreal• University of Toronto

– The Netherlands• University of Amsterdam

– Switzerland• CERN, Geneva

• Connectivity provided byCANARIE SURFnet ORION Netera BCnet RISQ

• Sources of funding: IGT (Canarie Directed Research Grant), ESTA (IST-2001-33182)

• Collaboration: CERN Openlab, the EU DataTAG project

GNEW 2004 4

International

Testbed Current IGT topology

GNEW 2004 5

International

Testbed Motivation

• IGT is a research network for next-generation e-science– Illustration of the application empowered

network concept (see Bill St. Arnaud’s panel at SC2003)

– Investigate emerging technologies in a demanding real life environment (see Wade Hong’s presentation at the CA*Net 4 Design Meeting May 26-27, 2003 )

– Real Time Farms for the ATLAS experiment at CERN (see Bryan Caron’s presentation at RISQ2003)

GNEW 2004 6

International

Testbed The ATLAS Experiment at CERN

• Large collaboration of about 2000 scientists for an experiment operational from 2007

• Main goals: the discovery of new particles and exploring physics beyond the Standard Model

• Challenging data collection and analysis systems– Online: 20 Gbps to the

event filter– Offline: 4 TB/day to

storage

GNEW 2004 7

International

Testbed Ethernet: from local to global

• Ethernet, a brand name for LAN technology– Original Ethernet: shared media, half-duplex,

distance limited by protocol– Modern Ethernet: point-to-point, full-duplex,

switched, distance limited by the optical components

• Cost effective !

GNEW 2004 8

International

Testbed Why native Ethernet long haul?• More than 90% of the Internet traffic originates on an Ethernet

LAN• Data traffic on the LAN increases due to new applications• Ethernet services with incremental bandwidth offer new

business opportunities to carriers– See IEEE Communications Magazine, Vol. 42, No. 3, March 2004, on

additional benefits for both the enterprise and the service providers

• Why not native Ethernet ?– Scalability, reliability, service guarantees …

• All of the above are active research areas

• Native Ethernet long haul connections can be used today as a complement to the routed networks, not as a replacement

GNEW 2004 9

International

Testbed 10 GE: a new Ethernet

• 10 GE – designed from the beginning for access to long haul networks– 40 km maximum distance specified by the standard …

– 1550nm lasers: optical amplifiers can be used to increase distance over dark fibre

• State of the art: 250 km demonstrated in Denmark by the EU ESTA project [see Mikkel Olesen’s presentation at NORDunet2003]

– what happens when you have to regenerate the signal ?• no signal-agnostic regenerators deployed

GNEW 2004 10

International

Testbed The 10 GE WAN PHY

• 10GE introduces a gateway from LAN to the WAN by means of the WAN PHY– Compatible with existing

WAN infrastructure• Transmission rate• Encapsulation

– Partial use of the management bits of the SONET/SDH frame

• Today’s WAN PHY modules use SONET-compliant optical components

WANWANPHY

LTE

LTE3R

3R

3R

OC192Router

10GE switch/router

LTE

RouterOC192

WANPHY

10GE switch/router

LTE

traditional

novel

GNEW 2004 11

International

Testbed Demo during ITU Telecom World '03

CiscoONS 15454Force10

E 600

Force10E 600

HPItanium-2

HPItanium-2

CiscoONS 15454

CiscoONS 15454

CiscoONS 15454

CiscoONS 15454

Ixia400T

IntelItanium-2

IntelXeon

Ixia400T

10GE WAN PHY 10GE LAN PHY OC192c

Ottawa Toronto Chicago Amsterdam Geneva

10 GE WAN PHY over an OC-192c circuit using lightpaths provided by SURFnet and CANARIE

9.24 Gbps using traffic generators

5.65 Gbps using TCP on PCs

6 Gbps using UDP on PCs

GNEW 2004 12

International

Testbed Results on the transatlantic 10 GE

Single stream UDP throughput Single stream TCP throughput

•Data rates are limited by the PC, even for our memory-to-memory tests•UDP uses less resources than TCP on high bandwidth-delay product networks

GNEW 2004 13

International

Testbed WAN PHY over DWDM

• Direct lambda access from the provider is required• The DWDM transceiver card as “LTE”

HP Itanium-2

10 GE WAN

DWDM

DWDM

Amsterdam Geneva

Force10 E600

10GE LAN

HPItanium-2

10 GE WAN10GE LAN

Ixia400T

Ixia400T

Force10 E600 HP

Itanium-2

IntelXeon

GNEW 2004 14

International

Testbed What next ?

• More 10 Gigabit Ethernet experiments– Extend the reach of our WAN PHY connection

• Adds more latency into the testbed, hence increased pressure on protocols

• Might allow us to identify technical showstoppers

– Disk-to-disk transfers– Comparative study of data transfer protocols

for 10 Gbps networks

GNEW 2004 15

International

Testbed What next ?

• Support for real-time access to remote computer farms during the ATLAS testbeam run in 2004– 1 Gbps connectivity is sufficient as a proof of

concept– Sites in Copenhagen, Cracow and Edmonton

will process in real time data acquired at the CERN testbeam

• This is part of the feasibility study for remote real time processing in ATLAS

GNEW 2004 16

International

Testbed Conclusions

• IGT demonstrated native 10 Gigabit Ethernet over lightpaths

• 10GE WAN PHY is the technology that enables inter-continental native Ethernet

• IGT will support remote real time applications, with emphasis on data collection and analysis

GNEW 2004 17

International

Testbed References

• Bill St. Arnaud – contribution to the “Strategies for Application-Empowered Networks” panel at SC2003

• Bryan Caron – “The GARDEN Project: A Testbed for High Bandwidth Real-Time Applications”, http://www.risq.qc.ca/risq2003-canw2003/ppt/23PM_Bryan_Caron.pdf

• Wade Hong – “CA*net 4 International Grid Testbed ”, http://lightpath.physics.carleton.ca/C4DesignIGTMay27.ppt

• Mikkel Olesen – “The use of dark fibre in Forskningsnttet”, the 21st NORUnet Conference, http://www.nordunet2003.is/smasidur/presentations/MikkelOlesen.ppt