Requirements

Mechanisms + Policies

API

Domain 1 Domain 2 Domain 32

BA

GNSA

LDG

1 3

GNSA GNSA

LDG

LDG

control plane control plane control plane

Ser

vice

pla

ne

ChicagoAmsterdam

Require fat unidirectional pipesTight QoS requirements (jitter, delay, data loss)

Simultaneous connectivity to multiple sitesMulti-domainDynamic connectivity hard to manageUnknown sequence of connections

Request to receive data from 1 hour up to 1 day Innovation for new bio-scienceArchitecture forces optimization of BW utilizationTradeoff between BW and storage

Network Issues

Personnel required at remote location

Remote instrument access (Radio-telescope)

N* previous scenarioAccess multiple remote DB

Copy from remote DB: Takes ~10 days (unpredictable)Store then copy/analyze

Point-to-Pointdata transfer of multi-TB data sets

Current MOPApplication Scenario

2 PB/YearComputational fluid dynamics

1 PB/yearMagnetic fusion

5 PB/YearPlasma physics

3 PB/YearNuclear physics

1 PB/yearSLAC (BaBar experiments)

10 PB/yearCERN LHC (Higgs boson search)

5 PB/yearRHIC (Quark-gluon plasma experiments)

<10 PB/yearCEBAF (Hadron structure experiments)

Estimated 2008 Data Generation

Processor

Performance

Traffic Growth

2x/12 months

x 16

20052010

2015

x 4

2020

x 32

2x/18 months

Data Transmission Plane

optical Control Plane

1 n

DB

1

n

1

n

Storage

Optical Control Network

Optical Control Network

Network Service Plane

Data Grid Service Plane

NRS

DTS

Compute

NMI

Scientific workflow

Apps Middleware

Resource managers

PRA

CreateReservationFromRequestSRA1

SRA2

SRA3

(ConstructProposals)

GetWindowsForRequest

ProposeReschedule

etc.

ConstructProposalForWindow

Fa

bric

UDP

ODINResources

Grid FTP

BIRN Mouse

Apps Middleware

TCP/HTTP

Grid

Layered

Arch

itecture

Lambda Data Grid

IP

Co

nn

ec

tivity

Ap

plic

atio

nR

es

ou

rce

Co

llab

ora

tive

BIRN Workflow

NMI

NRS

BIRN Toolkit

Lambda

Resource managers

DB

Storage Computation

Optical Control

WSRF

Optical protocols

Optical HW

OGSA

Availability: Abundant Optical Bandwidth

Requirements: Data-Intensive e-Science apps

Lambda Data Grid

Applications

Our emphasis

Grid work’s emphasis

Applications Middleware

Network Middleware

Network

• Terabit/s• 100Gb/s• 10Gb/s

1Gb/s

Fiber transmission

Edge computer limitations

Optical Control Network

Optical Control Network

Network Service Request

Data Transmission Plane

OmniNet Control PlaneODIN

UNI-N

ODIN

UNI-N

Connection Control

L3 router

L2 switch

Data storageswitch

DataPath

Control

DataPath Control

DATA GRID SERVICE PLANEDATA GRID SERVICE PLANE

1 n

1

n

1

n

DataPath

DataCenter

ServiceControl

ServiceControl

NETWORK SERVICE PLANENETWORK SERVICE PLANE

GRID Service Request

DataCenter

DWDM-RAM Service Control Architecture

Optical Control Network

Optical Control Network

Network Service Request

Data Transmission Plane

Optical Control PlaneOpticalControl

UNI-N

Optical Control

UNI-N

Connection Control

L3 router

L2 switch

Data storageswitch

DataPath

Control

DataPath Control

DATA GRID SERVICE PLANEDATA GRID SERVICE PLANE

1 n

1

n

1

n

DataPath

DataCenter

ServiceControl

ServiceControl

NETWORK SERVICE PLANENETWORK SERVICE PLANE

GRID Service Request

DataCenter

DWDM-RAM Service Control Architecture

From 100 Days to 100 Seconds

Lambda Data Grid - Globus Services

Fab

ric

SABUL

UDP

ODIN

OMNInetStorage Bricks

Grid FTP

GRAM

GSI

e-Science applications

Multidisciplinary Simulation

SOAP

TCP/HTTP

NRS

Storage Service

DTS

IP

Co

nn

ec

tivity

Ap

plicatio

nR

esou

rceC

olla

bo

rativ

e

GARA

Problem Solving Environment

Applications and Supporting Tools

Application Development Support

Common Grid

Services

LocalResources

Gri

d

Info

rmat

ion

S

ervi

ce

Un

ifo

rmR

eso

urc

eA

cces

s

Bro

keri

ng

Glo

bal

Q

ueu

ing

Glo

bal

Eve

nt

Ser

vice

s

Co

-S

ched

uli

ng

Dat

a C

atal

og

uin

g

Un

ifo

rm D

ata

Acc

ess

Co

mm

un

icat

ion

Se

rvic

es

Au

tho

riza

tio

n

Grid Security Infrastructure (authentication, proxy, secure transport)

Au

dit

ing

Fau

lt

Man

ag

emen

t

Mo

nit

ori

ng

Communication

ResourceManager

CPUs

ResourceManager

Tertiary Storage

ResourceManager

On-Line Storage

ResourceManager

Scientific Instruments

ResourceManager

Monitors

ResourceManager

Highspeed Data

Transport

ResourceManagernet QoS

Grid access (proxy authentication, authorization, initiation)

Grid task initiation

Collective Grid

Services

Fabric

Dat

a R

epli

cati

on

High performance computing and Processor memory co-allocationSecurity and Generic AAAOptical NetworkingResearched in other programlinesImported from the Globus toolkit

value

time

Window

value

time

Increasing value

time

Decreasing

value

time

Peak

value

time

Level

value

time

Asymptotic Increasing

value

time

Asymptotic Increasing

value

time

Step

Application Application

Services Services Services

data

control

data

control

Chicago Amsterdam

AAA

LDG LDGLDG

AAA AAA AAA

LDG

OMNInetOMNInetODIN Starligh

t

Starlight

Netherlight

Netherlight UvAUvA

ASTNASTNSNMPSNMP

Multi-EndPoint Communication

• Network Transfers Faster than Individual Machines– A Terabit flow? A 100Gbit flow? A 10Gbps flow w/ 1Gbps NIC’s

– Clusters are Cost-effective means to terminate Fast transfers

– Support Flexible, Robust, General N-to-M Communication– Manage Heterogeneity, Multiple Transfers, Data Accessibility

Uh-oh!

λData Receiver Data Source

FTP client FTP server

DMS NRM

Client App

Data Management Service

A

D

B

C

X

7:00-8:00

A

D

B

C

X

7:00-8:00

Y

Data service

Scheduling logic

Replica service

NMI /IF

Apps mware I/F

Proposal evaluation

NRS I/F

GT4 /IF

Data calc

DTS

Topology map

Scheduling algorithm

Proposal constructor

NMI /IF

DTS IF

Scheduling service

Optical control I/F

Proposal evaluator

GT4 /IF

Network allocation

Net calc

NRS

VisualizationX 1,000

Storage X 400Computation

X 500

Few-to-few 10Gbps connectivity with C=500, S=400, and V=1,000.

Will require budget of 100 trillion dollars a year

VisualizationX 1,000

Storage X 400Computation

X 500

Enabling new degrees of App/Net coupling

• Hybrid Optical Packet– Use ephemeral optical circuits to steer the herd of elephants (few to few)– Mice or individual elephants go through packet technologies (many to many)– Either application-driven or network-sensed; hands-free in either case– Other hybrid networks being explored (e.g., wireless + wireline)

• Application-engaged networks– The application makes itself known to the network– The network recognizes its footprints (via tokens, deep packet inspection)– E.g., storage management applications

• Workflow-engaged networks– Through workflow languages, the network is privy to the overall “flight-plan”– Failure-handling is cognizant of the same– Network services can anticipate the next step, or what-if’s– E.g., healthcare workflows over a distributed hospital enterprise

BIRN With the OptIPuter we are BIRN With the OptIPuter we are Addressing the Challenges of Large and Addressing the Challenges of Large and

Distributed DataDistributed Data

Each Brain is Big Data Each Brain is Big Data and Comparisons Must and Comparisons Must

be Made Between be Made Between Many!Many!

. ~5um

} 512 x 512x100,000

Invisible Nodes, Elements,

Hierarchical,Centrally Controlled,

Fairly Static

Traditional Provider Services:Invisible, Static Resources,

Centralized Management

Distributed Device, Dynamic Services, Visible & Accessible Resources, Integrated As Required By Apps

Limited Functionality,Flexibility

Unlimited Functionality,Flexibility

OptIPuter Paradigm Shift

fc *

Parallelism Has Come to Optical Networking (WDM)

Source: Steve Wallach, Chiaro Networks

“Lambdas”Parallel Lambdas Will Drive This Decade

The Way Parallel Processors Drove the 1990s

From: Smarr Talk “The Beginning of the Access Grid” April 15, 1999 www.jacobsschool.ucsd.edu/~lsmarr/talks/ACCESS.4.99_files/frame.htm

P-CSCFPhys. PCSCF

Session Convergence &

NexusEstablishment

End-to-endPolicy

DRAC Built-inServices

(sampler)

WorkflowLanguage

3rd PartyServices

AAA

Access

Value-AddServices

Sources/Sinks

Topology

Metro

Core

Proxy Proxy ProxyProxyProxy

P-CSCFPhys. P-CSCF

Proxy

Grid CommunityScheduler

•smart bandwidth management •Layer x <-> L1 interworking

•Alternate Site Failover

•SLA Monitoring and Verification •Service Discovery

•Workflow Language Interpreter

Bird’s eye View of the Service Stack

</DRAC>

<DRAC>

LegacySessions

(Management & Control Planes)

ControlPlane A

ControlPlane B

OAMOAMOAMPOAMOAMOAMPOAMOAMOAMPOAMOAM

OAMOAMOAMOAM

OAMOAM

How It Works: A Notional View

Admin.

Application

connectivity plane

virtualization plane

dynamic provisioning plane

Alert, Adapt,Route, Accelerate

Detect

supplyevents

eventssupply

AgileNetwork(s)

Application(s)

AAA

NE

from/to peering DRACs

demand

Negotiate

DRAC, portable SW

Routed IP NetworkRouted IP Network

GE

Customer A

CustomerAnetwork

CustomerAnetwork

Customer B

CustomerBnetwork

CustomerBnetwork

High-cap user High-cap user

GE

PP8600

PP8600

DRAC-driven Bypass in Action

VLAN XRouted IP

VLAN XRouted IP

Layer 1Bandwidth

Layer 1Bandwidth

ControlPlane

“DRAC”

UNI

VLAN YCloud Bypass

Flows differentiatedOn IP Subnet or port

Application Application

Services Services Services

Going multi-domain: the SC2004 Demonstrator

data

control

data

control

Chicago Amsterdam

• finesse the control of bandwidth across multiple domains

• while exploiting scalability and intra- , inter-domain fault recovery

• thru layering of a novel SOA upon legacy control planes and NEs

AAA

DRAC DRACDRAC

AAA AAA AAA

DRAC

OMNInetOMNInetODIN Starligh

t

Starlight

Netherlight

Netherlight UvAUvA

ASTNASTNSNMPSNMP

2nd Case Study: DataCenter•CPU + DATA + NET Orchestration•Impact of Virtualization on the End-to-End Session

Site 1 Site 2 Site 3

Horizontal IT IntegrationThe way a provider gainfully operates

for many paying customersover a geographical footprint

What Joe Smith thinks that he’s getting in exchange for a

monthly check to a provider

Joe Smith’sown Mainframeand good Apps

100 - 10,000 blades

vs.

Joe Smith’s Virtual Machines run herew/ Apps, Licenses (SAN not shown)

w/ right-sized bandwidth, 24x7, at 100% disaster-free Zip code

““Horizontal IT Integration” has multiple facets: Horizontal IT Integration” has multiple facets: Virtualization, SOA, Grid Computing, e-Utilities, Service GridsVirtualization, SOA, Grid Computing, e-Utilities, Service Grids

Virtual Compute PlaneVirtual Compute Plane

Virtualized Data-Centers w/ integrated resource control

AppVM

AppVM App

VM

Multi-domain Virtual Network, Security & Services PlaneMulti-domain Virtual Network, Security & Services Plane

Policies Policies Policies

Met

a-S

ched

ule

r

Mu

lti-

reso

urc

e C

oo

rdin

atio

n P

lan

e

Secure RouterNortel VR5000

L3 SwitchNortel ERS8600

L2 SwitchNortel ES5500

Metro OE GatewayNortel OM35/5/65

Application SwitchNortel AS2424SSL

User PlaneUser Plane

WS

Computing RM

WS

Device RM

WS

Storage RM

In Focus: Multi-Resource Coordination Plane

WS = Web ServicesRM = Resource Manager

• Instruments• Sensors• SCADA• RFID infr.

Met

a-S

ched

ule

r

Mu

lti-

reso

urc

e C

oo

rdin

atio

n P

lan

e

DRAC

WS

Execution Engines

DRAC

WS

DRAC

WS

Computation at the Right Place & Time!We migrate live Xen VMs, unbeknownst to

applications and clients, with dynamic cpu+data+net orchestration

Computation at the Right Place & Time!We migrate live Xen VMs, unbeknownst to

applications and clients, with dynamic cpu+data+net orchestration

Seattle

Netherlight

Amsterdam

NYC

Toronto

SC|2005

UvA

Starlight

Chicago

VMs

DynamicLightpaths

hitless remote rendering

The SC05 “VM Turntable” Demonstrator

The whole IT Industry is on a journey

Old World

Static

Silo

Physical

Manual

Application

New World

Dynamic

Shared

Virtual

Automated

Service

© GGF

WHAT ARE WEB SERVICES?• Web services are simple XML-based messages for

machine-machine messaging

–Web services don’t necessarily involve web browsers

–Think of web services as XML-based APIs

• Web services use standard internet technologies to interact dynamically with one another

–Well understood security model

–Loosely coupled

–Can be combined to form complex services–Open agreed standards connect disparate platforms

• Middleware based on web services has enjoyed tremendous success in the past five years

–eBay/PayPal, Amazon and Google all big users of web services

Google’s web service offerings:>Search Google’s eight billion web page database>Dictionary lookup

eBay’s usage of web services:>1 billion web service transactions per month>40% of listings now generated via web services

Web services rapidly becoming an essential part of many IT services in both B2B and B2C market categories

Example: Lightpath Scheduling

• Request for 1/2 hour between 4:00 and 5:30 on Segment D granted to User W at 4:00

• New request from User X for same segment for 1 hour between 3:30 and 5:00

• Reschedule user W to 4:30; user X to 3:30. Everyone is happy.

Route allocated for a time slot; new request comes in; 1st route can be rescheduled for a later slot within window to accommodate new request

4:30 5:00 5:304:003:30

W

4:30 5:00 5:304:003:30

X

4:30 5:00 5:304:003:30

WX

☺

4:30 5:00 5:304:003:30

W

4:30 5:00 5:304:003:30

X

4:30 5:00 5:304:003:30

WX

A

B

C

Scheduling Example - Reroute • Request for 1 hour between nodes A and B between

7:00 and 8:30 is granted using Segment X (and other segments) is granted for 7:00

• New request for 2 hours between nodes C and D between 7:00 and 9:30 This route needs to use Segment E to be satisfied

• Reroute the first request to take another path thru the topology to free up Segment E for the 2nd request. Everyone is happy

A

D

B

C

X7:00-8:00

A

D

B

C

X7:00-8:00

Y

Route allocated; new request comes in for a segment in use; 1st route can be altered to use different path to allow 2nd to also be serviced in its time window

☺

x.x.x.1

y.y.y.1

Optical cut-through

x.x.x.1

y.y.y.1Optical cut-through

e-Science example Application Scenario Current MOP Network Issues

Pt – Pt Data Transfer of Multi-TB Data Sets

Copy from remote DB: Takes ~10 days (unpredictable)Store then copy/analyze

Want << 1 day<< 1 hour, innovation for new bio-scienceArchitecture forced to optimize BW utilization at cost of storage

Access multiple remote DB

N* Previous Scenario Simultaneous connectivity to multiple sitesMulti-domainDynamic connectivity hard to manageDon’t know next connection needs

Remote instrument access (Radio-telescope)

Cant be done from home research institute

Need fat unidirectional pipesTight QoS requirements (jitter, delay, data loss)

Other Observations:• Not Feasible To Port Computation to Data• Delays Preclude Interactive Research: Copy, Then Analyze• Uncertain Transport Times Force A Sequential Process – Schedule Processing After Data Has Arrived• No cooperation/interaction among Storage, Computation & Network Middlewares•Dynamic network allocation as part of Grid Workflow, allows for new scientific experiments that are not possible with today’s static allocation