LambdaStation

LambdaStation

Phil DeMar Don Petravick

BNLOct. 25, 2004

10.07/2004 Demar, Petravick: Lambda Station 2f

FermilabOverview

• The Problem Space

• Functional Description of LambdaStation

• Components of LambdaStation

• Technical Approach

• Case Study

• Test Environment

• Project Outline

• Current Status

• Summary


FermilabThe Problem

• Very high (1-10 Gb/s) bandwidth research networks becoming available Typically not intended for general network traffic

Alternate paths for high impact data movement

Vision of the future: optical networks & lots of λ’s

• Not integrated with facility production LANs Production systems & storage facilities not directly

accessible Customized local access usually required…

WAN paths difficult to effectively use

Used mostly with prototypes or in demonstrations


Fermilab

• Proposed implementations don’t scale Multi-homed end system model breaks down with

multiple alternate paths Graceful failover capability missing or very complex

Virtual interface model difficult to administer & adds complexity on host systems

• Applications may need customization for each new technology or high-performance path

• A last mile problem between advanced WAN & facility production LANs…

The Problem (II)


Fermilab

Proposed Solution:

LambdaStation

• LambdaStation: an alternate path selection service Agent to coordinate use of alternate WAN paths: Dynamic configuration adjuster of local network devices Goal is per-flow granularity for alternate path selection

• LambdaStation is a research project A 2-3 yr US Dept of Energy, Office of Science Fermilab and Caltech In very early stages of development

Still in design phase Production network use is a ways off


Fermilab

How LambdaStation

Would Help• Advanced research networks

Offer alternate, high bandwidth WAN paths

May be a scheduled resource Typically demands customized

local access

• LambdaStation intended to: Facilitates application request for

high bandwidth WAN path Coordinates alternate WAN path

establishment Dynamically reconfigures LAN

forwarding of application’s dataLocal NetworkInfrastructure

Local NetworkInfrastructure

Internet

LambdaStationLambdaStation

Sto

rag

e s

yste

ms S

torag

e syste

ms

AdvancedResearch Network



Sto

rage system

sSto

rag

e sy

stem

s

Internet

AdvancedResearch Network


Fermilab

Components of LambdaStation

• A network path controller with modules interfacing to: Advanced WANs Local network infrastructure Peer LambdaStations Local applications &

middleware AA infrastructure

• Software on host systems to coordinate with LambdaStation

LambdaStation

Local networkequipment

ResourcePredictiveService

Advancednetwork

scheduling&

provisioning

VirtualOrganization(s)

Peerlambdastation

Layer 4: TCP, UDP, advanced stacks

Storage System(s)

Layer 5:direct fileaccess

Layer 5:filetransport

Layer 5:management(SRM)

Networkawareness

module

Middleware & Applications

Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork

operatingenvironment

Storage &Application Space

local demand module(s)

local steering service

WAN module

VO module


Fermilab

Host Interactions

• Path request primitives: Flow identification:

Source/dest address/port DSCP code point Possibly IP version

Transfer characteristics Bytes, peak rate, duration Schedule (range)

• Path status primitives: Allocated bandwidth Path setup status Path termination status

Normal or early abort…

LambdaStation



Advancednetwork

scheduling&

provisioning


Peerlambdastation


Storage System(s)




Networkawareness

module


Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork





WAN module

VO module


FermilabLAN Interactions• LAN device forwarding

reconfigurations Graceful alternate path

setup & teardown

• ACL modifications at site egress point

• Customizable interface to fit site specifics

• Non-objectives: QoS Traffic policing

LambdaStation



Advancednetwork

scheduling&

provisioning


Peerlambdastation


Storage System(s)




Networkawareness

module


Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork





WAN module

VO module


Fermilab

WAN Interactions

• Capability for multiple alternate WANs Customizable interface to fit

each WAN Provide abstract view for

applications

• Path reservation Resource availability Scheduling

• Path setup/teardown Graceful…

• Non-objectives: QoS Link efficiency

LambdaStation



Advancednetwork

scheduling&

provisioning


Peerlambdastation


Storage System(s)




Networkawareness

module


Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork





WAN module

VO module


Fermilab

Remote Site Interactions

• Establish path symmetry Exchange traffic selectors Coordinate WAN path

setup/teardown Ack LAN path

reconfiguration

• Verify path continuity Implies concurrent

connectivity via reserved & commodity network paths

Inform of: Path termination Traffic termination

LambdaStation



Advancednetwork

scheduling&

provisioning


Peerlambdastation


Storage System(s)




Networkawareness

module


Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork





WAN module

VO module


Fermilab

Authorization & Authentication

• Initial AA capability: LambdaStation internal Site-controlled

But some trust-level between sites

• Longer term goal: Utilize VO AA

infrastructure Raises site-level control

issues

LambdaStation



Advancednetwork

scheduling&

provisioning


Peerlambdastation


Storage System(s)




Networkawareness

module


Networkawareness

module

WANSpace

LocalNetworkSpace

Localnetwork





WAN module

AA module


Fermilab

Technical Approach

• Flow identification: Source/destination IP and DSCP code point tag Source/destination IP & Source/destination port #

• Local path forwarding reconfigurations: Based on policy routing ACLs

• Advanced WAN interface model: Site egress/ingress router to alternate path(s) Layer 2 site-to-site connections (vlan) or MPLS paths

• Transparent failover back to production path: Bandwidth limits or QBSS on fall-back traffic


FermilabUse Case1. FNAL SRM gets request to copy 3.5TB data set to CMS tier-2 site

- Transfer involves 3500 1GB files stored on 50 dCache systems

2. SRM initiates data transfer optimized for production network path3. SRM makes alternate path request to local LambdaStation4. LambdaStation:

- Checks authentication & authorization- Schedules & coordinates establishment WAN path- Coordinates with remote site LambdaStation for symmetrical path- Verifies alternate WAN path continuity- Reconfigures local LAN for selective flow forwarding- Notifies SRM that alternate path is established

5. SRM makes parallelism changes to re-optimize data transfer• SS Potentially shapes traffic to avoid congestion at pipe entrance.

6. Path reservation expires; LambdaStation gracefully tears path down- Remaining data movement traffic reverts to production network path- LambdaStation notifies SRM of alternate path teardown

7. SRM re-optimizes data transfer parallelism to complete the transfer


Fermilab

Test Environment

• LambdaStation Project Sites: FNAL (US-CMS Tier-1 Center) Cal Tech (US-CMS Tier-2 Site) Working to include UCSD as

well (US-CMS Tier-2 Site) CERN CMS Edge Computing

also planned

• LambdaStation Test WANs: US DOE UltraScience Net

Two 10 GE (OC192) lambda’s MPLS tunnels over DOE’s

ESnet Sub-10GE chunk out of ESnet

production network

FNALANL

tSunnyva

leFNAL

StarlightSunnyvale

Cal Tech

UltraScience Net (2 x 10 GE)

UCSD

UltraScience Net

MPLS path over ESnet

(<10 GE)

To CERN

(10

GE)


FermilabProject OutlineYear 1 LAN integration: Dynamic LAN reconfiguration

WAN integration: Interface to DOE UltraScience Net;Establish alternate path to Cal Tech

Application integration: SRM allocated bandwidth awareness ---------------------- ---------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------

Year 2 LAN integration: Automated LAN path setup/teardown, & scheduling; Peer LambdaStation coordination

WAN integration: Automated UltraScience Net WAN path setup/teardown & scheduling

Application integration: VO sensitivity; Adapt storage systems to schedule transfers to path avail.

---------------------- ---------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------

Year 3 LAN integration: Harden to production use quality

WAN integration: Full integration w/ UltraScience Net;

Application integration: Harden to production use quality


FermilabCurrent Status• A “Pre-LambdaStation” prototype exists:

Dynamically reconfigures LAN devices to forward select traffic over alternate paths

End system-applied DSCP tags for special-treatment flows

• Prototype work-arounds: Manual input of reservation parameters Static alternate WAN path – no setup required Asymmetric – no reverse path setup

• Results: Alternate traffic path established & torn down cleanly Path reroute & failback unnoticed by application


FermilabSummary• LambdaStation’s role in data-intensive science:

Dynamically connect production end-systems to advanced high-performance wide-area networks

Bring production-use systems to advanced networks Bring advanced networks to production-use applications

• LambdaStation’s status: Prototyping has demonstrated feasibility of dynamically

reconfiguring local network paths for specific data flows

Much work remains to be done for complete proof-of-concept

It’s a research project…

Documents

LambdaStation