View
217
Download
1
Tags:
Embed Size (px)
Citation preview
George Em KarniadakisDivision of Applied Mathematics
The CRUNCH group: www.cfm.brown.edu/crunch
Cross-Site Simulations on the TeraGrid
spectral elements Micro / Nano-fluidics parallel computing
Grand-Challenge Problem 1: Turbulence – Drag crisis(Tightly-Coupled Problem)
• Turbulence – Last frontier in classical physics
• Climate, environment, transport, energy,…
• Re=300,000 (CPU ~ Re3) requires 20 Billion DOFs
• Memory 4 TBytes
Wave Propagation in a Model of the Arterial Circulation
(Data of 55 main arteries from J.J. Wang and K. Parker, 1997)
Grand-Challenge Problem 2: Human Arterial Tree(Loosely-Coupled Problem)
First ParallelTeraGrid
Paradigm
NCSA IA64SDSC IA64
in-sitecommunication
Cross-sitecommunication
in-sitecommunication
TG Site TG Site
Whole flowDomain
All-to-all
-5/3
DNS versus Experiments: max Re=10,000
DNSExperiments
(Rockwell, 2004)
Energy Spectrum
Black – simulation
Blue - experiment
RMS velocity
Turbulence: Single-Site Performance
Fixed problem size Fixed workload
• PSC: Compaq Alpha EV68, 1 GHz
• 300 Million DOFs, 2-level MPI
• MPICH-G2 and MPI perform similarly (SDSC/IA-64)
• Half processors from NCSA, half from SDSC
• Intel IA-64 processors (Itanium-2, 1.5 GHz)
• Slow-down factor 1.5SDSC TG
NCSA TGFFT Matrix
transposition
Turbulence: Cross-Site Performance
Fixed problem size Fixed workload
P(t)
W1
W2
Ascending aorta
U(t)
Inflow conditions U(t)
P(t)
Thoracic aorta
Femoral
P(t)
U(t) W1
W2
U(t)
Tibial
P(t)
Outflow conditions(Peripheral resistance)
1D Model – Sherwin et al. / Imperial College
Platelet Aggregation in Arterioles and Platelet Aggregation in Arterioles and VenulesVenules
FLOW
Parameters: Vessel diameter - 50 µm, vessel length - 400 µm, blood velocity - 100 µm/s,platelet diameter - 3 µm, platelet concentration - 300000/mm3, platelet density - 1.03 fluid densitySimulation time - 28 s
venules
platelet aggregate
Growth Rate vs. Blood VelocityGrowth Rate vs. Blood Velocity
Experiments: Begent and Born, Nature, Vol. 227, No. 5261, pp. 926-930, 1970
Second Parallel TeraGrid Paradigm
Multiscale Simulation of Arterial Tree
Arterial-Tree: Cross-Site Performance
(Homogeneous Network)
• Three arteries; 4 Million DOFs per artery
• 1CPU/node on ANL; 2CPUs/node on NCSA/SDSC
• No slown-down, full scalabilitySDSC TG
ANL TG
NCSA TG
Fixed problem size Fixed workload
SDSC TG
NCSA TGPSC TG
Arterial-Tree: Cross-Site Performance
(Heterogeneous Network)
•PSC connects to TG via
application gateway (qsockets)
•Two arteries per site
•PSC proc:2 GF vs 6 GF IA-64
New Unique Capability
• Potentially unlimited salability; Enabling technology
– Integrate “real and virtual” in projects like:– Digital human, digital ocean, digital space, …
• Predictability and Uncertainty – Stochastic simulations– Prediction vs. Postdiction– Risk-based/Reliability-based design– Sensitivity analysis – steering of experiments
(e.g., DDDAS concept)
• Inverse Problems– Engineering design– Biomedical sciences– Geological/Climate Modeling
What Users Need
• Debuggers for TG (a la TotalView)• New topology-aware parallel algorithms• Sustained network/cluster performance• TG visualization capability• Middleware
– Robust MPICH-G2– Co-scheduling– Network & Globus diagnostics– Authentication/Security – often in conflict
• Consultants/Referees with TG-Expertise