400G Demonstrator for ISC ‘13 - t-systems-sfr.com · 400G Demonstrator for ISC ‘13 HPCN Workshop, Braunschweig, 7. Mai 2013 Wolfgang Wünsch, Technische Universität Dresden

1

400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13400G Demonstrator for ISC ‘13

HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013HPCN Workshop, Braunschweig, 7. Mai 2013

Wolfgang Wünsch, Technische Universität Dresden

Eduard Beier, T-Systems International

–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13

Agenda

� Partner

� Purpose

� Project Structure

� Topology

� Turbine Development

� Climate Computing

� Service Recipient Relations

� Data Path

� The Big Picture

� Project Lifetime

� Timeline

� DATE

� Test items

just click on the just click on the just click on the just click on the

hyperlinked topics! hyperlinked topics! hyperlinked topics! hyperlinked topics!

–public – E. Beier/ W. Wünsch 400G Demonstrator für ISC’13 3

Partner

Back to Back to Back to Back to AgendaAgendaAgendaAgenda


Purpose

The purpose of the project is: to demonstrate, that bandwidth beyond 100GBit/s is feasible and useful



Project Structure


Project Project Project Project BoardBoardBoardBoardProf. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. Prof. Dr. A. Bode / Prof. DrDrDrDr .W. Nagel .W. Nagel .W. Nagel .W. Nagel

Dr. A. KlugeDr. A. KlugeDr. A. KlugeDr. A. Kluge F. SchneiderF. SchneiderF. SchneiderF. Schneider Prof. Dr. W. GentzschProf. Dr. W. GentzschProf. Dr. W. GentzschProf. Dr. W. Gentzsch R. R. R. R. WienekeWienekeWienekeWieneke

M. M. M. M. ZappolinoZappolinoZappolinoZappolino Dr. A. GeigerDr. A. GeigerDr. A. GeigerDr. A. Geiger M. M. M. M. RoosenRoosenRoosenRoosen M. FuchsM. FuchsM. FuchsM. Fuchs

A. A. A. A. ClaubergClaubergClaubergClauberg T. T. T. T. WeselowskiWeselowskiWeselowskiWeselowski Jan Jan Jan Jan HeichlerHeichlerHeichlerHeichler

Project ManagementProject ManagementProject ManagementProject ManagementE.BeierE.BeierE.BeierE.Beier W. WünschW. WünschW. WünschW. Wünsch n.nn.nn.nn.n....

WP1WP1WP1WP1PerformancetTestsPerformancetTestsPerformancetTestsPerformancetTests

Andy GeorgiAndy GeorgiAndy GeorgiAndy Georgi

System Performance

Metering

WP2WP2WP2WP2Parallel Parallel Parallel Parallel FilesystemsFilesystemsFilesystemsFilesystems

Klaus GottschalkKlaus GottschalkKlaus GottschalkKlaus Gottschalk

Filesystem

Optimizing

WP3WP3WP3WP3Server & StorageServer & StorageServer & StorageServer & Storage

Beier/WünschBeier/WünschBeier/WünschBeier/Wünsch

Server & Storage

Project

Management

WP4WP4WP4WP4TransportTransportTransportTransport

MaskosMaskosMaskosMaskos / Mayer/ Mayer/ Mayer/ Mayer

PlanningPlanningPlanningPlanning / / / / EngineeringEngineeringEngineeringEngineering

WDM Project

Management

WP5WP5WP5WP5Layer 2/3Layer 2/3Layer 2/3Layer 2/3

Daniel Daniel Daniel Daniel NowaraNowaraNowaraNowara

Router Project

Management

WP7WP7WP7WP7ApplicationsApplicationsApplicationsApplications

Ferdinand Ferdinand Ferdinand Ferdinand JamitzkyJamitzkyJamitzkyJamitzky

Applications

Project

Management

WP6WP6WP6WP6SDN & NFVSDN & NFVSDN & NFVSDN & NFV

Ralf BraunRalf BraunRalf BraunRalf Braun

SDN & NFV &

Security

WP8WP8WP8WP8Public RelationsPublic RelationsPublic RelationsPublic Relations

Udo SchäferUdo SchäferUdo SchäferUdo Schäfer

Project

Marketing


WP1: Performance Tests

• Performance of subsystems (e.g. storage) and total performance measurements

•Feedback for subsystem optimization

•Conformance to measurement standards

•Input for publications

•

•WP lead: Andy GeorgiWP lead: Andy GeorgiWP lead: Andy GeorgiWP lead: Andy Georgi

Back to Back to Back to Back to Project StructureProject StructureProject StructureProject Structure


WP1: Targets


3x700G 3x700G 3x700G 3x700G FlashFlashFlashFlash

ServerServerServerServer Router Router Router Router 6 6 6 6 GByteGByteGByteGByte/s/s/s/s 5 5 5 5 GByteGByteGByteGByte/s/s/s/s

∼∼∼∼7000 7000 7000 7000 corescorescorescores

IB FDR IB FDR IB FDR IB FDR NetworkNetworkNetworkNetwork

7 7 7 7 G

Byte

GBy

teG

Byte

GBy

te/s/s /s/s


12x 12x 12x 12x ServerServerServerServer

RouterRouterRouterRouter72GByte/s72GByte/s72GByte/s72GByte/s 60GByte/s60GByte/s60GByte/s60GByte/s



84G

Byte

/s84

GBy

te/s

84G

Byte

/s84

GBy

te/s

55550GByte/s0GByte/s0GByte/s0GByte/s

Per ServerPer ServerPer ServerPer Server

TotalTotalTotalTotal


WP2: Parallel File System

•Planning, roll out, optimization and operation of the Parallel File system in coordination with other WPs and partners

•Configure and parameterize the Parallel File System (e.g. TCP buffers)

•Coordinate the communication between Clusters, File System and Network (IP Concept)


•

•WP lead: Klaus GottschalkWP lead: Klaus GottschalkWP lead: Klaus GottschalkWP lead: Klaus Gottschalk



WP3: Server & Storage & IB & etc

•Planning, roll out, optimization and operation of the server, storage and Infiniband infrastructure in coordination with other WPs and partners


•

•WP lead : WP lead : WP lead : WP lead : ProjektProjektProjektProjekt Management Management Management Management



WP4: Transport

•Planning, roll out, optimization and operation of the fiber and WDM infrastructure in coordination with other WPs and partners


•

•WP lead: Stefan WP lead: Stefan WP lead: Stefan WP lead: Stefan MaskosMaskosMaskosMaskos (Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)(Planning) / Heinz Mayer (Technology)



WP5: Layer 2/3

•Planning, roll out, optimization and operation of the Router infrastructure in coordination with other WPs and partners

•Input for publication

•

•WPWPWPWP----LeiterLeiterLeiterLeiter: Daniel : Daniel : Daniel : Daniel NowaraNowaraNowaraNowara



WP6: SDN & NFV & Security

• attend SDN and NFV approaches

• set up a security concept in coordination with the partners

•Implement that concept

• input for publications

•WP lead Ralf Braun (TWP lead Ralf Braun (TWP lead Ralf Braun (TWP lead Ralf Braun (T----Labs)Labs)Labs)Labs)



WP7: Applications

•Coordination of the application teams


•

•WP lead: Ferdinand WP lead: Ferdinand WP lead: Ferdinand WP lead: Ferdinand JamitzkyJamitzkyJamitzkyJamitzky



WP8: Public Relations

•Coordinate partners and activities towards optimum project marketing

•Coordinate press release activities

•Produce and attend project PR material (flyer, articles, etc)

•Coordinate ISC booth activities (flyer, logo, sessions, poster, give aways, etc)

•Coordinate the ISC application demonstration (incl. Internet access)

•

•WP lead: Udo WP lead: Udo WP lead: Udo WP lead: Udo SchäferSchäferSchäferSchäfer



100G Sachsen operational

100G Sachsen planning

400G Demonstrator p

Chemnitz

Topology

10GbE for Demonstrator

Computing Center

Euro Industriepark

München

DT PoP


Turbine Development


� Cooperation with DLR

� Workflow Demonstration

� Preprocessing

� Solver 1

� Solver 2

� Postprocessing

� Turbine model calculation with n Eigenmodes and m Phase Angels

Details:Details:Details:Details:

� Data volume: ~ 1TB

� Overall Workflow:

� Multitude of independent simulation runs (HTC).

� Simulations running on HPC resources at different sites.

� Every simulation produces input data for subsequent simulations.

� Subsequent simulations again run at different sites.

Thus to avoid knock-on delays in workflow execution data instantly should be availableat different sites!

GPFS:GPFS:GPFS:GPFS:

� Adopted feature: Active File Management (AFM) and Stretched Cluster

� Cross site data replication allows running simulations without prior copying

� Implicit data consistent backup via AFM data replication

Turbine Development: Benefits of GPFS Usage on 400G

Back to Back to Back to Back to Turbine Development Turbine Development Turbine Development Turbine Development

400G:400G:400G:400G:

� Possible job distributions on HPC resources:

� n * m jobs (n Eigenmodes, m phase angles)

� a * b jobs on cluster (a parallel jobs running on c cores, b serial jobs) with a * b ≥ n * m

� Bandwidth requirements: bandwidth = ��∙��∆�

, where� �� = Avg. file size written per job

� �� = No. of running in parallel (Here: corresponds to a)

� ∆� = Avg. time for disk access


No. cores

time

a = 6

n * m = 28 ≥ a * b = 30

∆t

Solver 1

Solver 2

b = 5

240min


400G: 400G: 400G: 400G: BandwidthBandwidthBandwidthBandwidth requirementsrequirementsrequirementsrequirements forforforfor different different different different jobjobjobjob distributiondistributiondistributiondistribution setupssetupssetupssetups

� Extreme/HTC setup with a = n * m = 300, b = 1:

� Assuming jobs all writing within 15min to disk an avg. file size of 150MB (i.e. write peak):

� Required bandwidth: 400GBit/s400GBit/s400GBit/s400GBit/s

� Required machine size: > 19200 Cores (when single jobs run on 64 cores)

� „Gentle“ setup with a = 50, b = 6:

� Assuming jobs having an avg. runtime of 240min continously writing 150MB ofdata to disk to represent runtime differences over larger values b:

� Required bandwidth: 4GBit/s4GBit/s4GBit/s4GBit/s

� Required machine size: > 3200 Cores (when single jobs run on 64 cores)




Chemnitz

Turbine Development Setup

1. Preprocessing @ DSI 1. Preprocessing @ DSI 1. Preprocessing @ DSI 1. Preprocessing @ DSI

2. Flow Model 2. Flow Model 2. Flow Model 2. Flow Model Calculation Calculation Calculation Calculation (Solver 1) (Solver 1) (Solver 1) (Solver 1)

3. Flow Model 3. Flow Model 3. Flow Model 3. Flow Model Calculation Calculation Calculation Calculation (Solver 2)(Solver 2)(Solver 2)(Solver 2)

4. 4. 4. 4. PostprocessingPostprocessingPostprocessingPostprocessing @ DSI @ DSI @ DSI @ DSI



Chemnitz

Turbine Development & GPFS

Parallel Distributed Parallel Distributed Parallel Distributed Parallel Distributed

File SystemFile SystemFile SystemFile System

GPFSGPFSGPFSGPFS


� Order 30 different Order 30 different Order 30 different Order 30 different modelsmodelsmodelsmodels areareareare usedusedusedused worldwideworldwideworldwideworldwide

� Experiments Experiments Experiments Experiments withwithwithwith thesethesethesethese modelsmodelsmodelsmodels produce10s produce10s produce10s produce10s ofofofofPBytesPBytesPBytesPBytes

� 100s 100s 100s 100s ofofofof PbytesPbytesPbytesPbytes ofofofof datadatadatadata needneedneedneed totototo bebebebe comparedcomparedcomparedcomparedbetweenbetweenbetweenbetween multiple multiple multiple multiple sitessitessitessites worldwideworldwideworldwideworldwide

� Movement Movement Movement Movement ofofofof datadatadatadata shouldshouldshouldshould bebebebe withinwithinwithinwithin monthsmonthsmonthsmonths ****

Transfer Rate

Time totransport 1 PB of Data

10 Mbps ~ 27 years

1 Gbps ~ 97 days

100 Gbps ~ 23 hours* Otherwise the questions will be forgotten ;-) Statistics taken from: „BER Network Requirements Workshop”,

LBNL report LBNL-4089E 2010, P 33. Recommended Reading

Climate Computing

ExtremelyExtremelyExtremelyExtremely High High High High BandwidthBandwidthBandwidthBandwidth RequirementsRequirementsRequirementsRequirements

‘‘‘‘VeryVeryVeryVery Big Data‘Big Data‘Big Data‘Big Data‘



Climate Computing Application Setup

Folder 1Folder 1Folder 1Folder 1CMIPCMIPCMIPCMIP

Folder 2Folder 2Folder 2Folder 2CMIPCMIPCMIPCMIP

FederationFederationFederationFederation

PreallocationPreallocationPreallocationPreallocation

Model postModel postModel postModel post----processing and processing and processing and processing and analysisanalysisanalysisanalysisVisualisiationVisualisiationVisualisiationVisualisiation @ ISC ’13 @ ISC ’13 @ ISC ’13 @ ISC ’13

LeipzigLeipzigLeipzigLeipzig

Folder Folder Folder Folder 3333CMIPCMIPCMIPCMIP

Back to Back to Back to Back to Climate Climate Climate Climate ComputingComputingComputingComputing


CCA & GPFS & iRODS

GPFS and/or GPFS and/or GPFS and/or GPFS and/or

Global Namespace Global Namespace Global Namespace Global Namespace

iRODSiRODSiRODSiRODS

Back to Back to Back to Back to Climate Climate Climate Climate ComputingComputingComputingComputing


Service Recipient Relations

Distributed FoldersDistributed FoldersDistributed FoldersDistributed Folders

Distributed Folders Distributed Folders Distributed Folders Distributed Folders

ServiceServiceServiceService RecipientRecipientRecipientRecipient

Federation, Federation, Federation, Federation, PreallocationPreallocationPreallocationPreallocation

Research ClientResearch ClientResearch ClientResearch Client

PREP & POST on PREP & POST on PREP & POST on PREP & POST on Cloud ResourcesCloud ResourcesCloud ResourcesCloud Resources

CalculationCalculationCalculationCalculation

Climate ComputingClimate ComputingClimate ComputingClimate Computing

TRACE on HPC TRACE on HPC TRACE on HPC TRACE on HPC ResourcesResourcesResourcesResources

TRACE on HPC TRACE on HPC TRACE on HPC TRACE on HPC ResourcesResourcesResourcesResources

Turbine DevelopmentTurbine DevelopmentTurbine DevelopmentTurbine Development

Client evaluating Client evaluating Client evaluating Client evaluating results, e.g. TECPLOTresults, e.g. TECPLOTresults, e.g. TECPLOTresults, e.g. TECPLOT

5T5T5T5T 5T5T5T5T

5T5T5T5T

5T5T5T5T

5T5T5T5T

5T5T5T5T

Distributed FoldersDistributed FoldersDistributed FoldersDistributed Folders 5T5T5T5T



400GBit/s Data Path

Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel (4x100GbE)(4x100GbE)(4x100GbE)(4x100GbE)

LinkLinkLinkLinkSerSerSerServerververver

StorageStorageStorageStorage

GPFS GPFS GPFS GPFS FilesystemFilesystemFilesystemFilesystem


RouterRouterRouterRouterRouterRouterRouterRouter RouterRouterRouterRouterRouterRouterRouterRouter12x 12x 12x 12x ServerServerServerServer

12x4012x4012x4012x4012x4012x4012x4012x40GbEGbEGbEGbE



GBitGBitGBitGBit/s/s/s/s400 400 400 400

GBitGBitGBitGBit/s/s/s/s





12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

TUD Cluster


IB FDRIB FDRIB FDRIB FDR10 10 10 10 NetworkNetworkNetworkNetwork

12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

1010 1010

12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

1010 1010


The Big Picture

Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption



LinkLinkLinkLink

SerSerSerServerververver


GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System


FirewallFirewallFirewallFirewall∼∼∼∼200 200 200 200 corescorescorescores

1x10 1x10 1x10 1x10 1x10 1x10 1x10 1x10 GbEGbEGbEGbE



RouterRouterRouterRouter17x 17x 17x 17x ServerServerServerServer


RouterRouterRouterRouter

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2


17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x2T 17x2T 17x2T 17x2T DiskDiskDiskDisk


RouterRouterRouterRouter

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

GBitGBitGBitGBit/s/s/s/s400 400 400 400

GBitGBitGBitGBit/s/s/s/s






TUD Cluster RZG Cluster


12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

1010 1010

12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

1010 1010


12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

12xI

B 12

xIB

12xI

B 12

xIB

FDR

FDR

FDR

FDR

SGI SGI SGI SGI ClusterClusterClusterCluster

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE


ConnectionConnectionConnectionConnection RZG Router RZG Router RZG Router RZG Router –––– RZGRZGRZGRZG----WDM and WDM and WDM and WDM and Connection TUD RouterConnection TUD RouterConnection TUD RouterConnection TUD Router–––– TUD WDMTUD WDMTUD WDMTUD WDM

Type 2 x LC (100GBaseLR4)

length TUD: 10m; RZG: 10m

Volume 4 each (8 total)

7750 7750 7750 7750 SR12ESR12ESR12ESR12E7750 7750 7750 7750

SR12ESR12ESR12ESR12E

7750 7750 7750 7750 SR12SR12SR12SR127750 7750 7750 7750 SR12SR12SR12SR12

1830 1830 1830 1830 PSSPSSPSSPSS

4x10

04x

100

4x10

04x

100

GbE

GbE

GbE

GbE

4x10

04x

100

4x10

04x

100

GbE

GbE

GbE

GbE

Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch

400G WDM 400G WDM 400G WDM 400G WDM Super Channel Super Channel Super Channel Super Channel

LinkLinkLinkLinkWDM TerminalWDM TerminalWDM TerminalWDM Terminal

AmplifierAmplifierAmplifierAmplifier

1830 1830 1830 1830 PSSPSSPSSPSS

4x10

04x

100

4x10

04x

100

GbE

GbE

GbE

GbE

4x10

04x

100

4x10

04x

100

GbE

GbE

GbE

GbE

coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 coherent Super Channel (2 x 16QAM@ 50 GHzGridGHzGridGHzGridGHzGrid / 2 x 200 / 2 x 200 / 2 x 200 / 2 x 200 GBitGBitGBitGBit/s) /s) /s) /s)

OLA

+ O

LA+

OLA

+ O

LA+

DG

ED

GE

DG

ED

GE

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

OLA

70km

70km

70km

70km 70

km70

km70

km70

km

70km70km70km70km 70km70km70km70km 70km70km70km70km 70km70km70km70km70km70km70km70km70km70km70km70km70km70km70km70km

400 G WDM Super Channel

Back to Back to Back to Back to Big PictureBig PictureBig PictureBig Picture


ConnectionConnectionConnectionConnection TUD Server TUD Server TUD Server TUD Server –––– TUD IBTUD IBTUD IBTUD IB----Switch Switch Switch Switch andandandandConnectionConnectionConnectionConnection RZG Server RZG Server RZG Server RZG Server –––– RZG IB SwitchRZG IB SwitchRZG IB SwitchRZG IB Switch

Type MPO (Infiniband FDR (56GBit/s))

length TUD: 10m, RZG: 10m

Volume 12 each side (24 total)

Infiniband Connections

MellanoxMellanoxMellanoxMellanox ConnectConnectConnectConnect----IB IB IB IB Volume:12on each side (24total)Volume:12on each side (24total)Volume:12on each side (24total)Volume:12on each side (24total)


MellanoxMellanoxMellanoxMellanox active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)


ConnectionConnectionConnectionConnection TUD Server TUD Server TUD Server TUD Server –––– TUD Router TUD Router TUD Router TUD Router and and and and ConnectionConnectionConnectionConnection RZG Server RZG Server RZG Server RZG Server –––– RZG RouterRZG RouterRZG RouterRZG Router

Type MPO (40GBaseSR4)

length TUD: 10m; RZG: 10m

Volume 24x10m MellanoxMellanoxMellanoxMellanox Connect X3 Connect X3 Connect X3 Connect X3 Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)

MellanoxMellanoxMellanoxMellanox active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)active Cable (incl. QSFP)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)Volume: 12 on each side (24 total)

40GbE Connections

AlcatelAlcatelAlcatelAlcatel----Lucent 3Lucent 3Lucent 3Lucent 3----port 40GbE IMM port 40GbE IMM port 40GbE IMM port 40GbE IMM Volume: 4 @ RZG Volume: 4 @ RZG Volume: 4 @ RZG Volume: 4 @ RZG

AlcatelAlcatelAlcatelAlcatel----Lucent 6Lucent 6Lucent 6Lucent 6----port 40GbE IMM port 40GbE IMM port 40GbE IMM port 40GbE IMM (no picture) (no picture) (no picture) (no picture) Volume:2@TUDVolume:2@TUDVolume:2@TUDVolume:2@TUD



IBM IBM IBM IBM iDataPlexiDataPlexiDataPlexiDataPlex dx360 M4 dx360 M4 dx360 M4 dx360 M4 (Volume: 12@RZG)(Volume: 12@RZG)(Volume: 12@RZG)(Volume: 12@RZG)

Bull Bull Bull Bull NovaScaleNovaScaleNovaScaleNovaScale R460 F3 R460 F3 R460 F3 R460 F3 (Volume: 12@TUD) (Volume: 12@TUD) (Volume: 12@TUD) (Volume: 12@TUD)

Server



ConnectionConnectionConnectionConnection Router Router Router Router –––– WDM WDM WDM WDM

Type LC singelmode(100GBaseLR10)

length 10m

Volume 2


AlcatelAlcatelAlcatelAlcatel----Lucent 2Lucent 2Lucent 2Lucent 2----port 100GbE IMM port 100GbE IMM port 100GbE IMM port 100GbE IMM Volume: 3 @ TUD Volume: 3 @ TUD Volume: 3 @ TUD Volume: 3 @ TUD

AlcatelAlcatelAlcatelAlcatel----Lucent 1Lucent 1Lucent 1Lucent 1----port 100GbE IMM port 100GbE IMM port 100GbE IMM port 100GbE IMM (no picture) (no picture) (no picture) (no picture) Volume:4@RZGVolume:4@RZGVolume:4@RZGVolume:4@RZG


ConnectionConnectionConnectionConnection TUD Router TUD Router TUD Router TUD Router –––– TUD 10GbE ClusterTUD 10GbE ClusterTUD 10GbE ClusterTUD 10GbE Cluster

Type LC duplex multimode (10GBaseSR)

length ?

Volume 17



Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12Alcatel Lucent 7750 SR12

@RZG@RZG@RZG@RZG

Alcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 EAlcatel Lucent 7750 SR12 E

@TUD@TUD@TUD@TUD



Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32Alcatel Lucent 1830 PSS 32




EMCEMCEMCEMC2222 XtremSFXtremSFXtremSFXtremSF 700GB SLC700GB SLC700GB SLC700GB SLC

Volume: 36 Volume: 36 Volume: 36 Volume: 36 PCIePCIePCIePCIe cards on each side (72 total)cards on each side (72 total)cards on each side (72 total)cards on each side (72 total)


closingclosingclosingclosing

WSWSWSWS21.6.21.6.21.6.21.6.----????

37

Project Lifetime

Demonstrator SetupDemonstrator SetupDemonstrator SetupDemonstrator Setup Getting through Getting through Getting through Getting through Test Item List TILTest Item List TILTest Item List TILTest Item List TIL

ISCISCISCISC28.128.128.128.1---- 15.6. 15.6. 15.6. 15.6. KickoffKickoffKickoffKickoff

WSWSWSWS



Press Press Press Press ReleaseReleaseReleaseRelease

Timeline Rev. C

Setup Server, Storage, Setup Server, Storage, Setup Server, Storage, Setup Server, Storage, InfinibandInfinibandInfinibandInfiniband Setup GPFSSetup GPFSSetup GPFSSetup GPFS

7.6.

7.6.

7.6.

7.6.

final final final final PerfomancePerfomancePerfomancePerfomance TestsTestsTestsTests

12.6

.12

.6.

12.6

.12

.6.

16.6

.16

.6.

16.6

.16

.6.

RfSRfSRfSRfS WDM WDM WDM WDM & IP & IP & IP & IP

Performance Performance Performance Performance Tests Tests Tests Tests donedonedonedone RfSRfSRfSRfS GPFSGPFSGPFSGPFS

Setup WDM, RouterSetup WDM, RouterSetup WDM, RouterSetup WDM, Router

DATE DATE DATE DATE

Applications Applications Applications Applications final final final final configconfigconfigconfig

RfSRfSRfSRfS Tunnel, ServerTunnel, ServerTunnel, ServerTunnel, Server

RfSRfSRfSRfSApplicationsApplicationsApplicationsApplications


29.5

29.5

29.5

29.5

.

no Application Tests possibleno Application Tests possibleno Application Tests possibleno Application Tests possible

GPFS optimizationGPFS optimizationGPFS optimizationGPFS optimization

CW19CW19CW19CW19 CW20CW20CW20CW20 CW21CW21CW21CW21 CW22CW22CW22CW22 CW23CW23CW23CW23 CW24CW24CW24CW24

We are hereWe are hereWe are hereWe are here

10.6

.10

.6.

10.6

.10

.6.

20.5

20.5

20.5

20.5

.

14.6

.14

.6.

14.6

.14

.6.

Stop Stop Stop Stop DATEDATEDATEDATE


Demonstrator Application Test Environment

DATE

ObjectiveObjectiveObjectiveObjective

•3 weeks between RfS IP and the begin of the ISC is by far not enough time to get two high sophisticated applications running @400G

•Therefore the support of the applications as early as possible is an integrated part of the project; the application teams get access as soon possible on new building blocks of the ‘big picture’

1. April 1. April 1. April 1. April ---- 17. May 17. May 17. May 17. May



DATE Phase 1


LinkLinkLinkLink




Firewall / EncryptionFirewall / EncryptionFirewall / EncryptionFirewall / Encryption GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System

7750 7750 7750 7750 SR12SR12SR12SR12



7750 7750 7750 7750 SR12 SR12 SR12 SR12

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE



1. April 1. April 1. April 1. April ---- 29. April 29. April 29. April 29. April

Back to Back to Back to Back to TimelineTimelineTimelineTimeline


DATE Phase 2


LinkLinkLinkLink





3xGPFS 3xGPFS 3xGPFS 3xGPFS ServerServerServerServer

7750 7750 7750 7750 SR12SR12SR12SR12



7750 7750 7750 7750 SR12 SR12 SR12 SR12

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE






29. April 29. April 29. April 29. April ---- 6. May 6. May 6. May 6. May






DATE Phase 3


LinkLinkLinkLink






7750 7750 7750 7750 SR12SR12SR12SR12



7750 7750 7750 7750 SR12 SR12 SR12 SR12

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE




TUD Cluster





23xI

B 23

xIB

23xI

B 23

xIB

FDR

FDR

FDR

FDR

23xI

B 23

xIB

23xI

B 23

xIB

FDR

FDR

FDR

FDR

7. May 7. May 7. May 7. May ---- 16. May 16. May 16. May 16. May






Test Items

ObjectiveObjectiveObjectiveObjective

•The Demonstrator is a great opportunity for getting valuable findings in networking , HPC, virtualization and other fields

•the project schedule just allows some synthetic performance tests and the implementation of the applications

•Therefore the partners have agreed to set up a test item list where each partner is invited to contribute proposals

Back to Back to Back to Back to Project Project Project Project LifetimeLifetimeLifetimeLifetime

21.6.21.6.21.6.21.6.----????21.6.21.6.21.6.21.6.----????


Test Item List

NFV (TSI)NFV (TSI)NFV (TSI)NFV (TSI)


LoadbalancerLoadbalancerLoadbalancerLoadbalancer / Bundling / Performance / / Bundling / Performance / / Bundling / Performance / / Bundling / Performance / CoSCoSCoSCoS / FCAPS (T/ FCAPS (T/ FCAPS (T/ FCAPS (T----Labs) Labs) Labs) Labs)

21.6.21.6.21.6.21.6.----????21.6.21.6.21.6.21.6.----????

GPFS Network (TSIGPFS Network (TSIGPFS Network (TSIGPFS Network (TSI----SfRSfRSfRSfR) ) ) ) SDN (TUD) SDN (TUD) SDN (TUD) SDN (TUD)

RDMA over Ethernet RDMA over Ethernet RDMA over Ethernet RDMA over Ethernet


Network Functions Virtualisation (NFV)

Back to Back to Back to Back to Test Item ListTest Item ListTest Item ListTest Item List


Example NFV Use Case

ESX ESX ESX ESX ClusterClusterClusterClusterFirewallFirewallFirewallFirewall FirewallFirewallFirewallFirewall

Customer Customer Customer Customer BBBB

VLAN 2VLAN 2VLAN 2VLAN 2VLAN 1VLAN 1VLAN 1VLAN 1

Each customer configures Each customer configures Each customer configures Each customer configures his own FW entityhis own FW entityhis own FW entityhis own FW entity

FirewallFirewallFirewallFirewall

VLAN 1VLAN 1VLAN 1VLAN 1

CustomerCustomerCustomerCustomer AAAA




InternetInternetInternetInternet

Router/Switch Router/Switch Router/Switch Router/Switch

Firewall VMFirewall VMFirewall VMFirewall VM

Standby Firewall VMStandby Firewall VMStandby Firewall VMStandby Firewall VM



NFV & ESX Test Setup

ESX ESX ESX ESX Cluster Cluster Cluster Cluster

ClientClientClientClient

40

40

40

40

GbE

GbE

GbE

GbE40

40

40

40

G

bEG

bEG

bEG

bE

Router/Switch Router/Switch Router/Switch Router/Switch

Firewall VMFirewall VMFirewall VMFirewall VM

Standard HW Standard HW Standard HW Standard HW (2xE5(2xE5(2xE5(2xE5----2670 + 2670 + 2670 + 2670 + 128G RAM)128G RAM)128G RAM)128G RAM)

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

40 40 40 40

GbE

GbE

GbE

GbE

ClientClientClientClient

40

40

40

40

GbE

GbE

GbE

GbE40

40

40

40

G

bEG

bEG

bEG

bE



Demonstrator NFV Setup

Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch LinkLinkLinkLink


StorageStorageStorageStorageFirewall / Encryption / Firewall / Encryption / Firewall / Encryption / Firewall / Encryption /

CompressionCompressionCompressionCompressionGPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System





7750 7750 7750 7750 SR12SR12SR12SR12



7750 7750 7750 7750 SR12E SR12E SR12E SR12E

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE




7750 7750 7750 7750 SR12SR12SR12SR12

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE






3xIB

3x

IB

3xIB

3x

IB

FDR

FDR

FDR

FDR

1010 1010

3xIB

3x

IB

3xIB

3x

IB

FDR

FDR

FDR

FDR

1010 1010


3xIB

FD

R3x

IB F

DR

3xIB

FD

R3x

IB F

DR

1x1001x1001x1001x1001x1001x1001x1001x100GbEGbEGbEGbE3xFEC3xFEC3xFEC3xFEC 3x403x403x403x403x403x403x403x40

GbEGbEGbEGbE3x403x403x403x403x403x403x403x40GbEGbEGbEGbE 3xFEC3xFEC3xFEC3xFEC





1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE



ScheduledScheduledScheduledScheduled for: 21.6.for: 21.6.for: 21.6.for: 21.6.----5.7.?5.7.?5.7.?5.7.?ObjectivesObjectivesObjectivesObjectives

•Is 5GByte/s Firewall / Encryption / Compression with standard hardware (2xE5-2670 & PCIe3.0) feasible?•Even hypervisored?•What is the impact for the application?

NFV Objectives & Comments

CommentsCommentsCommentsComments

•In ‘real life’ network functions like encryption, firewalling and compression can become very important , even in HPC environments•There is no need for 400GBit/s; 100GBit/s is equivalent



Loadbalancer / Bundling / Performance / CoS / FCAPS Setup



GPFS Network





7750 7750 7750 7750 SR12SR12SR12SR12



7750 7750 7750 7750 SR12E SR12E SR12E SR12E

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

2x10

0 2x

100

2x10

0 2x

100

GbE

GbE

GbE

GbE

DATE Cluster 1

DATE Cluster 2


17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE

17x1

0 17

x10

17x1

0 17

x10

GbE

GbE

GbE

GbE



7750 7750 7750 7750 SR12SR12SR12SR12

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE







3xIB

3x

IB

3xIB

3x

IB

FDR

FDR

FDR

FDR

1010 1010

3xIB

3x

IB

3xIB

3x

IB

FDR

FDR

FDR

FDR

1010 1010


3xIB

FD

R3x

IB F

DR

3xIB

FD

R3x

IB F

DR



3x40

3x

40

3x40

3x

40

GbE

GbE

GbE

GbE

3x40

3x

40

3x40

3x

40

GbE

GbE

GbE

GbE

Router / SwitchRouter / SwitchRouter / SwitchRouter / Switch

LinkLinkLinkLink


StorageStorageStorageStorageFirewall / Encryption / Firewall / Encryption / Firewall / Encryption / Firewall / Encryption /

CompressionCompressionCompressionCompression

GPFS Files SystemGPFS Files SystemGPFS Files SystemGPFS Files System





1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE

1x10

1x

10

1x10

1x

10

GbE

GbE

GbE

GbE



ScheduledScheduledScheduledScheduled for: 21.6.for: 21.6.for: 21.6.for: 21.6.----5.7.?5.7.?5.7.?5.7.?ObjectivesObjectivesObjectivesObjectives

•Wie verhält sich GPFS,wenn die Daten nicht zwischen 2, sondern 3 Sites (Option 4 Sites, Hub be TUD) gespiegelt werden? (Server Überlast?)•Was passiert, wenn z.B eine Site mit geringerer Bandbreite angebunden ist? “Kommt das GPFS noch nach”?

Test Item GPFS Network


•There is no need for 400GBit/s; 100GBit/s is equivalent



Test Item SDN


* Aufsetzen einer virtuellen SDN Umgebung zwischen ZIH & RZG mittels vSwitch* Wünschenswert wäre die Einbindung von aktiven Netzwerkelementen mit OpenFlowUnterstützung (bspw. auch Barracudas SDN Gateway)* Gegenüberstellung verschiedener verfügbarer OpenFlow Controller (Beacon, Floodlight, FlowER, OpenDaylight, ...)* Zeitraum: 1 Monat, kann aber parallel zu anderen Untersuchungen laufen

Da vermutlich nicht besonders viel Zeit zur Verfügung stehen wird, glaube ich das dies bereits sehr ambitioniert ist. Ich muss dann schauen wieviele Controller aufgesetzt und getestet werden können. Aber ich würde die Umgebung so aufbauen das ich auch nach dem 400G Showcase noch damit arbeiten kann.



Test Item RDMA over Ethernet


- Analyse vorhandener Protokolle für RDMA über Ethernet- Vergleich mit RDMA über InfiniBand- Performanceanalyse und -optimierung innerhalb einer 40GbE Testumgebung- Gegenüberstellung der Ergebnisse mit FDR InfiniBand



Thank You!Thank You!Thank You!Thank You!

Documents

400G Demonstrator for ISC ‘13 - t-systems-sfr.com · 400G Demonstrator for ISC ‘13 HPCN Workshop, Braunschweig, 7. Mai 2013 Wolfgang Wünsch, Technische Universität Dresden