Institut du développement et des ressources ...spscicomp.org/wordpress/wp-content/uploads/2013/05/voury-IDRIS... · 1000000 10000000 100000000 Performance (Gflops) Evolution des

www.idris.fr Institut du développement et des ressources en informatique scientifique

IDRIS Site Update Pascal Voury, User Support Team

Institut du développement et des ressources en informatique scientifique

SPXXL / ScicomP – Lugano, May 2013

2 Institut du développement et des ressources en informatique scientifique

IDRIS Location



History of Supercomputing at IDRIS


Vorführender

Präsentationsnotizen

Workhorse ? Betes de somme.



History of Supercomputing at IDRIS

• Parallel scalar systems

• Cray T3D (1995) • Cray T3E (1996) • IBM SP3 (2001) • IBM P4 (2002)

• IBM P4+ (2003) • IBM BG/P (2008)

• IBM P6 (2009)

IBM BG/Q IBM x3750

Vector systems

Cray C98 (1993) Cray C94 (1994)

Fujitsu VPP300 (1997) NEC SX-5 (2000) NEC SX-8 (2006)

RIP Feb 2012













0,1

1

10

100

1000

10000

100000

1000000

10000000

100000000 Pe

rfor

man

ce (G

flops

)

Evolution des performances

Numéro 1

Numéro 500

IDRIS - somme

Earth Simulator

Slide provided by M.A. Foujols, IPSL, CNRS




Turing, IBM BG/Q


• 4 racks • 64 nodes per I/O node, • Everything is then proportional : 65TB memory, 65536 cores, 836 Tflops

etc.

• 2.2 TB disc shared with the x3750 (BW theoretically overloaded by 25 I/O nodes : 50GB/s) in 5 DDN SFA10K cabinets.


Ada, IBM x3750-M4


• 332 nodes with 4 Sandy Bridge E5-4650 8 cores @ 2.7 GHz • 28 nodes have 256GB memory, all the others have 128GB • Roughly 49 TB memory and 233Tflops • 2 nodes for interactive login, with discs • Plus 4 nodes x3870-M5 Westmere @ 2.67 GHz with 1 TB memory each

(and discs) for pre- and post-processing • GPFS 3.5, LoadLeveler 5.1, poe 1.2.12 • Mellanox InfiniBand FDR10 with a 648 ports switch and a second level of

36 ports switches (each node has 2 links to 2 switches)

• New for us : diskless nodes, optimization of the memory requirement for the OS image. Same HW, different SW stack. And another different set for post-processing.


Infrastructure


Vorführender


4 racks DDN


The good, the bad, the ugly


• From a User Support point of view : • The good : BG/Q pretty stable, as BG/P was. Surprised to learn IBM would

not correct the bugs in the software stack provided. • The « not so good » for BG/Q: hardware problems not detected by IBM

(NaNs, QCD code); change of I/O performance strategy vs. BG/P. • The bad for x3750 :

− RDMA engine halting the whole configuration : seems solved by Mellanox expertise for 2 weeks.

− Latency of support for Intel problems. Lack of experience on our side (Power for 12 years!). For example : how to limit the RSS memory taken by an OpenMP job ?

• The ugly for x3750: poe environment mandatory for performance, on our Intel platform. Could be OK, but we still have bugs with poe that we don’t have using Intel MPI (Buring Issue).

Vorführender


switch to DATAGRAM (UD) mode instead of CONNECTED (CM) mode. OFED 1.5.3, CM better with scalability issues (may to many). OFED 2.0, UD x2


E5 4650 internal architecture :



Political Developments


• IDRIS is not buying its own computers for the CNRS any more : GENCI does.

GENCI means Grand Equipement National pour le Calcul Intensif. Owned for 49 % by the French State represented by the Ministry for Higher Education and Research, for 20 % by CEA, 20 % by CNRS, 10 % by the Universities and 1% by INRIA. Created in 2007, GENCI provides funding and assumes ownership. Also promotes the organization of an European HPC area and participates to its achievements; GENCI is the french representative in PRACE.

• IBM did not promote its Power architecture, • No clear visibility yet on BG’s future, • Why would GENCI still buy an IBM for an Intel based computer ?

Vorführender


Catherine Riviere was appointed last year Chair of PRACE’s Council (for the next 2 years). She has succeeded Prof. Dr. Achim Bachem, Chairman of the Board of Directors of Germany’s Jülich Research Centre


Future Technical Developments


• Archive : robot is fine. Do we need a new design for our system? − HSM on one of the supercomputers − « Classical » design, because of limited financial enveloppe, with a disc

cache as big as possible; SSD discarded because of the price • Currently used as backup for result files in a batch; should evolve to a pure

archive system ? − 2.2 Po disc WORKDIR on the computers − Increase capacity, even at the cost of increased latency.


Grand Challenges on Ada


• DRAKKAR, Climatology : NEMO ocean model, 5 Mh (4000 cores)

ORCA 12 domain with it 3600 subdomains




• DEUS-PUR, Astrophysics : 4 Mh (64 to 9000 cores)




• SELTRAN, Molecular Dynamics : custom GROMACS, 1.5 Mh (64 cores) • LIQSIM, ab-initio Molecular Dynamics : CP2K, 0.3 Mh (512 cores)

Non-aqueous ionic solution Gromacs patched with PLUMED plug-in performance


Grand Challenges on Turing


• PrecLQCD, lattice QCD : 31.5 Mh (4 racks) • StabMat, QCD & QED, proton weight : 30 Mh (2 racks to 4 racks, Juelich) • BigDFT, ab-initio ion Li batteries: 16.6 Mh




• MesoNH, Tiwi Island « Hector » storm : 13 Mh (1 rack, Global Array I/Os)

Vorführender


Comparison with airborne LIDAR. Mesh from 800 to 100m, all realistic. Impact at a larger scale of a realistic simulation.




• ZoomBHA, astrophysics (Black Hole Accretion): 12 Mh

Vorführender


Zooming in from galaxy to BH for short periods of time (1M years). More turbulence and chaos than expected, general models have to change.




• GYSELA, Tokamak plasmas : 11 Mh (100 000 threads, 4 racks) Fluctuations of electrostatic potential when turbulence starts in plasma

Vorführender


ITER. 95% efficiency weak scaling btw 4 rack and half a one.




• MHDTURB, RAMSES magnetohydrodynamics code : 11Mh (2 racks)

Vorführender


Again astrophysics on accretion discs.




• APAFA, AVBP LES combustion : 10 Mh (2 racks, I/O perf. problems)




• ECOPREMS, LES combustion with stratification : 9.3 Mh

400 million tetrahedras

50 million tetrahedras


Our typical Workload


2012 figures for the decomissioned computers, very few changes (except political choices) • X3750 : 246 projects, 1100 individual users, 60 Mh allocated for 2013.

26 Million hours on Power6 in 2012

Vorführender


99% availability.


Our typical Workload


• BG/Q : 90 projects, 400 users, 297 Mh allocated for 2013.

245 Million hours on BG/P in 2012

Vorführender


290 Mh allocated, 99% availability. Fluid mechanics from 70 to 120Mh in a year (less QCD).



• Thank you for your attention.

• Questions ?

Documents

Institut du développement et des ressources ...spscicomp.org/wordpress/wp-content/uploads/2013/05/voury-IDRIS... · 1000000 10000000 100000000 Performance (Gflops) Evolution des