Terapaths: MPLS based Data Sharing Infrastructure for Peta Scale LHC Computing Bruce Gibbard and Dantong Yu USATLAS Computing Facility DOE Network Research

Terapaths: MPLS based Data Sharing Infrastructure for Peta Scale LHC Computing

Bruce Gibbard and Dantong YuBruce Gibbard and Dantong Yu

USATLAS Computing FacilityUSATLAS Computing Facility

DOE Network Research PI MeetingDOE Network Research PI Meeting

September 15-17, 2004September 15-17, 2004

B. Gibbard & D. YuB. Gibbard & D. Yu DOE Network Research PI Meeting

FNAL September 15-17, 2004

2

Terapaths Topics

Introduction: what is the Terapaths projectIntroduction: what is the Terapaths project

BNL network configuration and upgrade planBNL network configuration and upgrade plan

WAN usage at BNLWAN usage at BNL

Related projects at BNL: Grid AA and SRM/dCache Related projects at BNL: Grid AA and SRM/dCache

Program goals and detailsProgram goals and details

Work plans and scheduleWork plans and schedule

Summary and current statusSummary and current status



3

This project will investigate the integration and use of MPLS based This project will investigate the integration and use of MPLS based

differentiated network services in the ATLAS data intensive differentiated network services in the ATLAS data intensive

distributed computing environment as a way to manage the network distributed computing environment as a way to manage the network

as a critical resource; as a critical resource;

The program intends to explore network configurations from The program intends to explore network configurations from

common shared infrastructure (current IP networks) to dedicated common shared infrastructure (current IP networks) to dedicated

optical paths point-to-point, using MPLS/QoS to span the optical paths point-to-point, using MPLS/QoS to span the

intervening possibilities.intervening possibilities.

The Collaboration includes:The Collaboration includes: Brookhaven National Laboratory (US ATLAS Tier 1, ESNet)

Univ. of Michigan (US ATLAS Candidate Tier 2 Center, Internet2,

UltraLight)

What is Terapaths ?



4

BNL Campus Network and Immediate Update Plan

The USATLAS/RHIC Computing The USATLAS/RHIC Computing Facility is attached to access layer Facility is attached to access layer routers.routers.

Campus network consists of Cisco Campus network consists of Cisco 6509 Series Switch, MPLS support.6509 Series Switch, MPLS support.

Cisco PIX 535 Firewall. (1Gbps)Cisco PIX 535 Firewall. (1Gbps) Will be replaced by firewall

service blade for CISCO 6500 series. (5G bps), Sep/Oct 2004

Core router is connected to ESnet Core router is connected to ESnet Juniper M10 router. Juniper M10 router.

WAN connection will be upgrade to WAN connection will be upgrade to OC 48 (2.5Gbps) in Oct/Nov 2004 time OC 48 (2.5Gbps) in Oct/Nov 2004 time frame.frame.



5

Network Traffic Monitoring at GridFtp server and routers



6

Network Utilization

USATLAS is doing Grid enabled DC2 production and RHIC is sending USATLAS is doing Grid enabled DC2 production and RHIC is sending

physics data to the remote collaborators.physics data to the remote collaborators. Sustain USATLAS data transfer BNL DC2 sites at ~ 25MB/sec last month.

Goal: sustain data transfer BNL CERN at ~ 45MBytes/sec in OCT’ 04.

Network Monitoring, performance testing (iperf, GridFtp).Network Monitoring, performance testing (iperf, GridFtp).

BNL currently offers only best effort network service.BNL currently offers only best effort network service.

We periodically fill up OC-12 connection: intensive contentions over We periodically fill up OC-12 connection: intensive contentions over

limited network bandwidth give unhappy users.limited network bandwidth give unhappy users.

Network resource has to be systematically allocated and managed to Network resource has to be systematically allocated and managed to

deliver the most efficient, effective overall system!!!deliver the most efficient, effective overall system!!!



7

Related Projects: BNL Grid AA and SRM/dCache

GUMS (Grid User Management System) is a Grid identity mapping GUMS (Grid User Management System) is a Grid identity mapping services: Grid credential to local credential.services: Grid credential to local credential. Part of Privilege project: a joint project between USATLAS/USCMS. GUMS is in production for RHIC at BNL since May/2004. We are transforming GUMS into a service which the gatekeeper can contact

directly: a preliminary implementation was completed. Need extension to authorize network resource.

Storage Resource Managers (SRM) is to provide dynamic space Storage Resource Managers (SRM) is to provide dynamic space allocation and file management on Grid SE. It uses GridFtp for file allocation and file management on Grid SE. It uses GridFtp for file moving.moving. BNL USATLAS deployed two flavor of SRMs: Berkeley SRM and dCache. Capable of inter-operating with each other via web services interface. Provide access to USATLAS users. Experience and their characteristics are documented.



8

Project Goal and Objectives

The primary goal of this project is to investigate the use The primary goal of this project is to investigate the use of this technology in the ATLAS data intensive distributed of this technology in the ATLAS data intensive distributed computing environment. In addition we intend to:computing environment. In addition we intend to:

Develop expertise in MPLS based QoS technology which will be important to ATLAS and the LHC community more generally.

Dedicate fractions of the available WAN bandwidth via MPLS to ATLAS Tier 1 data movement, RHIC data replications to assure adequate throughput and limit their disruptive impact upon each other.

Enhance technical contact between the ATLAS tier 1 at BNL and its network partners including the Tier 0 center at CERN, potential ATLAS Tier 2’s and other members of the Grid3+ (OSG-0) community of which it is a part.



9

Proposed Prototype/Primitive Infrastructure

GridFtp & SRM

MPLS PathESnet

Network resource manager

MPLS requests

Traffic IdentificationTCP syn/fin packages,

addresses, port #

Grid AA

Network Usage Policy

Translator

MPLS Bandwidt

h Requests

& Releases

OSCARSINGRESS

Monitoring Direct MPLS

/Bandwidth Requests

SE

Second/Third yearSecond/Third year



10

Work Plans

Terapaths envisions a multiple year program to deliver a high-Terapaths envisions a multiple year program to deliver a high-performance, QoS enable network infrastructure for ATLAS/LHC performance, QoS enable network infrastructure for ATLAS/LHC computing. Each year will determine the following year(s)’s direction.computing. Each year will determine the following year(s)’s direction.

Phase I: Establish Initial Functionality (08/04 ~ 07/05).

Help to steer the direction of the following two phases.

Phase II:Establish Prototype Production Service (08/05 ~ 07/06).

Depends on the success of Phase 1.

Phase III: Establish Full Production Service, Extend Scope and Increase Functionality (08/06 ~ 07/07).

The level of service and its scope will depends on the available project funding and some additional resources.

Broaden deployment and capability to Tier2s, partners.



11

Establish MPLS paths with Initial Partners

SLAC

LBL

Sunnyvale

Albuquerque

Atlanta

New York

ChicagoPAIX

StarLight

El Paso

SLAC

LBNL

ESnet

BNL

DC

TWC

Abilene

Abilene

ANL FNAL

MPLS

CERN

MICH



12

Establish Initial Functionality

Milestones: Milestones:

Utilize network mock-up on Linux “routers” to test configuration and Utilize network mock-up on Linux “routers” to test configuration and capability.capability.

Setup MPLS path inside BNL campus network, connect GridFtp servers Setup MPLS path inside BNL campus network, connect GridFtp servers and SRM into MPLS based network.and SRM into MPLS based network.

Study Impact of MPLS on data transfer service and gain experience of Study Impact of MPLS on data transfer service and gain experience of using MPLS.using MPLS.

Study the behavior of MPLS path through firewall.Study the behavior of MPLS path through firewall.

Setup MPLS paths on routers from multiple vendors (Cisco and Juniper).Setup MPLS paths on routers from multiple vendors (Cisco and Juniper).

Test and verify MPLS paths between BNL and LBL, SLAC (network Test and verify MPLS paths between BNL and LBL, SLAC (network monitoring project), FNAL and CERN.monitoring project), FNAL and CERN.

Test and verify Inter Domain MPLS path between BNL and University of Test and verify Inter Domain MPLS path between BNL and University of Michigan.Michigan.



13

Contributions at Year 1

Build MPLS expertise in BNL: MPLS setup, configuration, Build MPLS expertise in BNL: MPLS setup, configuration,

maintenance and removal.maintenance and removal.

Learn the effectiveness and efficiency of MPLS and its Learn the effectiveness and efficiency of MPLS and its

impact to overall network performance: MPLS and non-impact to overall network performance: MPLS and non-

MPLS.MPLS.

Decide whether MPLS is useful to LHC physics.Decide whether MPLS is useful to LHC physics.

Document any lesson/experience learned from this project.Document any lesson/experience learned from this project.

Raise aware of that network resource can/should be Raise aware of that network resource can/should be

managed.managed.



14

Establish Prototype Production Service

Integrate Grid data transfer (GridFtp) into MPLS enabled network Integrate Grid data transfer (GridFtp) into MPLS enabled network service.service.

Effectively couple these network/data transfer services with Effectively couple these network/data transfer services with storage resources managed by SRM, have GridFtp/SRM storage resources managed by SRM, have GridFtp/SRM functionality in “beta”.functionality in “beta”.

Incorporate the resulting system into the ATLAS grid middleware.Incorporate the resulting system into the ATLAS grid middleware.

Build tools to provide basic authentication, authorization and Build tools to provide basic authentication, authorization and access control – Depend on funding, Rely on leverage. access control – Depend on funding, Rely on leverage.

Supply client interfaces which make this service available in a Supply client interfaces which make this service available in a manner transparent to any details of the underlying QoS/MPLS manner transparent to any details of the underlying QoS/MPLS traffic engineeringtraffic engineering

Leverage MPLS paths/VO level network monitoring services with Leverage MPLS paths/VO level network monitoring services with DWMI project to be developed at SLAC.DWMI project to be developed at SLAC.



15

Extend Scope and Increase Functionality of Prototype Service

Inter-network domain MPLS establishment, dynamically creating, adjusting of Inter-network domain MPLS establishment, dynamically creating, adjusting of sub-partitioning MPLS paths to meet time constrained network requirements.sub-partitioning MPLS paths to meet time constrained network requirements.

Create site level network resource manager for multiple VOs vying for limited WAN resource.

Provide dynamic bandwidth re-adjusting based resource usage policy and path utilization status collected from network monitoring (DWMI).

Leverage dynamic MPLS establishing services provided by OSCARS: the ESnet On-Demand Secure Circuits and Advance Reservation System.

Create user interface/web services for LHC data transfer applications to request network resource in advance.

Goal: to broaden deployment and capability to tier 1 and tier 2 sites, create services which will be honored/adopted by CERN ATLAS/LHC Tier 0.



16

Summary and Status

Summary: this project will prototype and deploy a QoS capable Summary: this project will prototype and deploy a QoS capable wide area networking service based on MPLS to support wide area networking service based on MPLS to support ATLAS/LHC data transfer.ATLAS/LHC data transfer.

Current Status:Current Status: A MPLS simulation testbed is being created in a private network

environment. Evaluate mechanisms to assign different labels to GridFtp data

transfers initiated by different VOs. Different GLOBUS_TCP_PORT_RANGE for multiple VOs, source IP

addresses and port numbers determine labels. Use IP CoS bits to assign labels at border router.

Open Science Grid authentication/Authorization Systems are being developed to provide access control to various resource. (GUMS software development)

Berkeley SRM and dCache/SRM were evaluated and deployed to interface BNL storage resource into Grid.

Documents

Terapaths: MPLS based Data Sharing Infrastructure for Peta Scale LHC Computing Bruce Gibbard and Dantong Yu USATLAS Computing Facility DOE Network Research