A Case Study on Multi-Component Multi-Cluster Interaction ...hpc.pnl.gov/conf/wolfhpc/2013/talks/atanasov.pdfWOLFHPC 2013 A. Atanasov, H.-J. Bungartz, K. Unterweger,T. Weinzierl, R

Technische Universität München

A Case Study on Multi-ComponentMulti-Cluster Interaction with an AMR Solver

WOLFHPC 2013

A. Atanasov, H.-J. Bungartz, K. Unterweger,T. Weinzierl, R. Wittmann

November, 18th 2013

A. Atanasov: A Case Study on Multi-Component Multi-Cluster Interaction with an AMR Solver

WOLFHPC 2013, November, 18th 2013 1


Outline

Motivation

Communication

Software Architecture

Results

Conclusion & Outlook




Motivation

IO IO IO

IO IO IOIO IO IO

IO IO IO

NM

M̂

• Multi-cluster interactions between non-monolithic applications

• Examples : Multi-Physics, simulation-postprocessing coupling, . . .

• M request sources, N data sources, M̂ destination nodes




Use Case

View

Model

Controller

• Simulation Model: Parallel AMR solver for hyperbolic PDEs [4]

• Visualisation: Query-based data retrieval [3]

• Controller : User located in VR environment




Outline

Motivation

Communication


Results





M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation

Simulation

Controller




M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation

Simulation

0

0

Controller




M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation

Simulation

0

0

1 1

Controller




M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation

Simulation

0

0

1 1

2

Controller




M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation

Simulation

0

0

1 1

2

3

Controller




Routing Information (0 − 3)




Merges : Greedy Merge Version(4 + 5)

M

SPoC

IO IO IO

IO IO IO

IO IO IO

IO

25%

10

IO

35%

60%

IO

10%10%

20%

N

M̂

Visualisation

Simulation

0

0

1 1

2

3

4

Controller

40%





M

SPoC

IO IO IO

IO IO IO

IO IO IO

IO

25%

10

IO

35%

60%

IO

10%10%

20%

N

M̂

Visualisation

Simulation

0

0

1 1

2

3

4

Controller

40%





M

SPoC

IO IO IO

IO IO IO

IO IO IO

IO

25%

10

IO

35%

60%

IO

10%10%

20%

N

M̂

Visualisation

Simulation

0

0

1 1

2

3

4

Controller

40%





M

SPoC

IO IO IO

IO IO IO

IO IO IO

IO

25%

10

IO

35%

60%

IO

10%10%

20%

N

M̂

Visualisation

Simulation

0

0

1 1

2

3

4

Controller

40%

• Global identification of the nodes holding most data• Merge data on identified nodes• Forward merged data to destinations




M

SPoC

IO IO IO

IO IO IOIO IO IO

IO IO IO

N

M̂Visualisation Simulation

0

0

1 1

2

3

4

Controller

4

4




Hierarchical Merge Version(4)

Level 0

Level 1

Level 2

Level 3

Query

Level 0

v0 v1 v2 v3 v4 v5

cutoff level

cutoff level

• Use a logical tree to merge data• Merge data up to given cutoff ℓ

• Destinations merge the remaining data fragments




Tree Merge, cont.

Properties

• Requires a logical master-worker hierarchy

• Fully asynchronous

• Additional memory requirements dependent on ℓ

• ℓ can be chosen dynamically

• Merge distributed between N and M̂




Outline

Motivation

Communication


Results






Scientific IDL

Proxygenerator

Component1 Component2

ASCoDT Framework

ASCODT Builder

Static Repository

DynamicRepository

CCAPorts

Static RepositoryAPI AbstractconfigurationAPI

Communicationspecificpartof theCCAPorts

Frameworkcalls CCAcodegenerator calls

Independentcomponentcalls

• A lightweighted distributed framework based on CCA [1]• Communication through BSD sockets or MPI Ports• IDE for component development [2]• Generate glue code from SIDL annotations




Outline

Motivation

Communication


Results





Results

1 2 4 8 16 32 640

1020304050607080

Level 3 full query

Avg. BandwidthMin.BandwidthMax. Bandwidth

[Mprocesses][Mprocesses]^

[Gbp

s]

1 2 4 8 16 32 6405

101520253035404550

Level 31/2 full query

Avg. BandwidthMin.BandwidthMax. Bandwidth

[Mprocesses][Mprocesses]^

[Gbp

s]

1 2 4 8 16 32 640

20

40

60

80

100

120

full query4096x4096

L3 TreeMerge TCPL3 TreeMergeMPIGreedy Merge TCP

[Mprocesses]

[Gbps]

^1 2 4 8 16 32 64

01020304050607080

1/2 full query4096x4096

L3 TreeMerge TCPL3 TreeMergeMPIGreedy Merge TCP

[Mprocesses]

[Gbps]

^

• Fully refined computational domain with 98 (6561 × 6561) cells• Two queries : full domain query, half domain query (4096 × 4096)




Outline

Motivation

Communication


Results






Enabling technologies

• On-demand data retrieval

• Destination-aware data routing

• Asynchronous partial merges

• Hide technical details through CCA

Outlook

• Autotuning approaches

• User-defined communication protocols

• Multi-physics applications




References I

R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. Kohn,L. McInnes, S. Parker, and B. Smolinski.Toward a common component architecture for high-performancescientific computing.In High Performance Distributed Computing, 1999. Proceedings. TheEighth International Symposium on, pages 115–124. IEEE, 1999.

A. Atanasov, H.-J. Bungartz, and T. Weinzierl.A toolkit for the code development in advanced computing.Technical Report TUM-I1330, 2013.




References II

A. Atanasov, M. Srinivasan, and T. Weinzierl.Query-driven parallel exploration of large datasets.In Large Data Analysis and Visualization (LDAV), 2012 IEEESymposium on, pages 23 –30, Oct. 2012.doi 10.1109/LDAV.2012.6378972.

K. Unterweger, T. Weinzierl, D. I. Ketcheson, and A. Ahmadia.Peanoclaw—a functionally-decomposed approach to adaptive meshrefinement with local time stepping for hyperbolic conservation lawsolvers.Technische Universität München, Technical Reports, 2013.



Documents

A Case Study on Multi-Component Multi-Cluster Interaction ...hpc.pnl.gov/conf/wolfhpc/2013/talks/atanasov.pdfWOLFHPC 2013 A. Atanasov, H.-J. Bungartz, K. Unterweger,T. Weinzierl, R