Sustainable DCI Operations scalable in

size, time, and complexity

Bernhard Schott, DGSI Coordinator, Platform Computing

Thijs Metsch, Senior Software Engineer, Platform ComputingGridKa, 08.09.2011, Karlsruhe, Germany

07.09.2011 2

Sustainable DCI Operations

• Agenda• The DGSI project• Interoperation, Federation, Operation• Implementation Technology• Standardization in OGF & IEEE• Rational for a federated DCI future

07.09.2011

The DGSI Project

• DGSI – D-Grid Scheduling Interoperabilitytargets:

• #1: Interoperation of multiple technologies

• #2: Federation of Grids & Clouds

• #3: Operation, sustainable & scalable

– Scalable in size: interoperability of “regions”

– Scalable in time: interoperability of “versions”

– Scalable in complexity: interoperability of “technologies and concepts”

07.09.2011 4

Interoperation of multiple technologies

MSS* Meta Sched.

Local Scheduler Local Scheduler Local Scheduler

GridWay*Meta Sched.

C3WSS *Meta Sched.5 different types

gLiteGlobusUnicore

*examples

(WMS)Meta Sched.

gLite

UnicoreGlobus

UnicoreGlobusgLite

07.09.2011 5

Federation Mode 1: Activity Delegation

Unicore Globus gLite-CE

Local Scheduler Local Scheduler Local Scheduler

AD AD

DFPDFPMSS MetaScheduler

C3WSS MetaScheduler

GridwayMetaSched

*DFP = DCI Federation Protocol

07.09.2011 6

Federation Mode 2: Resource Delegation

Unicore Globus gLite-CE

Local Scheduler Local Scheduler Local Scheduler

DFPDFPMSS MetaScheduler

C3WSS MetaScheduler

GridwayMetaSched

*DFP = DCI Federation Protocol

07.09.2011 7

Federation of Grids & Clouds

Clouds

Service Grids

Desktop Grids

CTRL

CTRL

MSMS

DGS

DGS

CTRL=Cloud controller

MS=Grid Meta-Scheduler

DGS=Desktop Grid Server

Combination of EDGeS bridge with DGSI Federation methods:

AD Activity Delegation

RD Resource Delegation

Dynamically configured by negotiation, SLAs & policies.

RD

PetaFlopsince 2008?

07.09.2011 8

DCI Operations: sustainable & scalable

• Operation, sustainable & scalable

– Scalable in size: interoperability of “regions”

– Scalable in time: interoperability of “versions”

– Scalable in complexity: interoperability of “technologies and concepts”

07.09.2011 9

Operation, sustainable & scalable: interoperability of regions

MWxy MWxy MWxy

Local Scheduler Local Scheduler Local Scheduler

DFPDFPNE-Europe SouthGridCentral-EU

*DFP = DCI Federation Protocol

07.09.2011 10

Operation, sustainable & scalable: interoperability of versions

Globus 4.2 Globus 5.0

Local Scheduler Local Scheduler

DFP

Globus 4.1

Local Scheduler

DFPMS-vers.1.2 MS-vers.5.0MS-vers.2.3

*DFP = DCI Federation Protocol

07.09.2011 11

Operation, sustainable & scalable: interoperability of versions

gLite 4 gLite 5

Local Scheduler Local Scheduler

DFP

gLite 3

Local Scheduler

DFPMS-vers.1.2 MS-vers.5.0MS-vers.2.3

*DFP = DCI Federation Protocol

Globus 4.1

Local Scheduler

MS-vers.1.2

07.09.2011 12

Interoperability of concepts: ISF & DGSI

Platform ISFShared computing infrastructure

HPC Computing Cluster

CERN compute center

HTC Computing

Resource Delegation

virtually moves resources

DFP

FederationBroker Instance

HPC Cloud Service

Grid

Globus 4.1

Local Scheduler

MS-vers.1.2

07.09.2011 13

Interoperability of concepts: ISF & DGSI

Platform ISFShared computing infrastructure

HPC Computing Cluster

CERN compute center

HTC Computing

Resource Delegation

virtually moves resources

DFP

FederationBroker Instance

Implementation Technology

Activity Delegation Resource Delegation

12.03.10 14DGSI: Delegation methods

Implementation Technology

Activity Delegation Resource Delegation

12.03.10 15DGSI: Delegation methods

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.16TORONTO 9/7/2011

The 4 reasons why Standards matter

Or why Standards* rock!Thijs Metsch, Senior Software Engineer

08-09-2011, GridKa School 2011, Karlsruhe

*Industrial, Programming language, SDOs, …

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.17

PORTABILITY

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.18

“Transport” / Format Definition

?

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.19

Inte

rests

Time

Funding Program Interests

Data / Methodology portability?

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.20

Technology

SOAP

SQL

NoSQL

REST

C/C++

Java

Ruby

Python

JavaScript

…

???

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.21

Compute power?

Source: http://www.fujitsu.com/img/KCOM/top/kcom-main-number1.jpg

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.22

INTEROPERABILITY

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.23

DUH??

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.24

…IT ISN'T EASY…

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.25

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.26

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.27

GLUE / BES / JSDL / …

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.28

# of Standards to solve a Prb. ???# o

f std

s

Complexity/Simplicity of your problem

?

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.29

# of Standards to solve a Prb. ???# o

f std

s

Complexity/Simplicity of your problem

?

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.30

http://xkcd.com/927/

http://nighthacks.com/roller/jag/entry/xkcd_on_standards

?

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.31

Interoperability is hard! Creating/Using a Standard is not necessary

going to solve the problem…

Protipp: Do it right!

How? Depends - there is no recipe (but learn from the past –

which standards and their processes are good which bad etc.)

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.32

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.33

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.34

INTEGRATION

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.35

Devices & location

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.36

STARTUP

(Some) Startups: plug’ it together?

HTTP

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.37

INNOVATION

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.38

INNOVATION

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.39

If technology thrills you…

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.40

…and you like working in groups…

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.41

…join a group and…

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.42

…create cool standards & technology.

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.43

Don’t forget: share it...

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.44

…and write it in you CV!

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.45

The time is now…

http://nighthacks.com/roller/jag/resource/StandardsPhases.html

Time

Inte

rrest

Time for

Standards

Copyright © 2011 Platform Computing Corporation. All Rights Reserved.46

Thank you…