Dr. Wolfgang Zierau (zierau@uni-muenster.de)Heinz-Hermann Adam (adamh@uni-muenster.de)

Integrity Superdome for Scientific ComputingApplications of High Performance and High Throughput Computing in physical sciences at the University of Münster

HP-CAST 12Madrid, Spain – May 10 - 13, 2009

Heinz-Hermann Adam

Who we are

• WWU Münster is one ofthe three majoruniversities in Germany•~ 40,000 students

•~ 5,000 scientists andstaff

•Over 100 fields of study

• The Natural SciencesDepartment is ~ ¼ of theuniversity•Major user and provider

of compute resources

•IT is a Volunteer drivenoperation•Not much dedicated staff

11.05.20092

Heinz-Hermann Adam

What we do

• Provide and maintainresources for students, scientists and staff in Biology, Chemistry andPhysics•~ 4,000 Computers

•~ 12,000 Users

• Compute resources•Scientific Computing

•SMP and Clusters

•Development Environment

•Desktop Applications•Windows

•Linux

•Mac OS

•File and Print Sharing

11.05.20093

Heinz-Hermann Adam

Problems needing high computational power

• Theoretical Chemistry•Quantum-Monte-Carlo

•Gaussian

• Theoretical Physics•Nano-Structure formation

•Non-linear physics

• Experimental Physics•Nano Technology

• Bioinformatics•BLAST

• Condensed Matter Theory•Microscopic theory of

lattice dynamics

•Carrier dynamics in semiconductors

• Geophysics•Fluid dynamics

• Nuclear physics•GRID-Computing

11.05.20094

Heinz-Hermann Adam

Montecito Superdome Installation 2006

• Replacement of Out-of-date VMS and Tru64 UNIX 4-way AlphaServers

• Moving away fromdatasilos

• Virtualization at theheart of everything

• High ThroughputComputing – crossplatform

• 4 Partition Superdome 32B running Linux andWindows

• Heading towards datapools with SAN storage

• EVA, BladeSystem c-Class, VMware VI 3

• Condor Pool of severalhundred PCs

11.05.20095

Heinz-Hermann Adam

Initial Configuration• 4 Hardpartitions• 8 Cellboards• 24 Itanium² Montecito

CPUs•Dual-Core•Hyperthreading

• 96 GB RAM•4 GB per Socket

• Per Partition I/O•Ethernet•SAN•SCSI•RAID

11.05.20096

Heinz-Hermann Adam

Basic Software

• Intel Compiler•Fortran

•C/C++

• Math Kernel Library

• Vtune Performance Analyzer

• Linux•Batchqueue

•Torque/Maui

• Windows•Batchqueue

•Condor

11.05.20097

Heinz-Hermann Adam

2008 Upgrade

• After 1 ½ year ofoperation

• No change topartitioning

• Upgrade of smallestLinux partition to RHEL 5 (XEN)•Exploring virtualization

without HP-UX/IntegrityVM or Virtuozzo

• Throughput of Batch partition is limited byavailable memory, not number or performanceof processors•Doubled the amount of

memory on that partition•Symmetrical for optimum

performance

11.05.20098

Heinz-Hermann Adam

Sample Calculations

• Localization of SurfacePolaritons at Nano-Surfaces•Laserbeam incident on a

Silver Surface•Defined surfacestructure:

known

•Surface with statisticalroughness: unknown

•Statistics: Large numberof samples is required

• Reflection of light isbeing „measured“•What is the effect of

excitations on such a nano surface?

11.05.20099

Illustration of the scattering geometry

Heinz-Hermann Adam

• Solution of coupled linear integral equations•Reduced Rayleigh equations

• Complex Matrices > 4,000 x 4,000 elements•For each element solve a surface integral with 10,000 –

20,000 nodes

11.05.2009

Computational Model

10

( ) ( ) ( )( )( )∑ ∫−

−=−

+++−−=

1 cos1sincos2

2

21

,N

Nn

bnc

irc

i

ssedrR

π

π

θαθωθθω

θθ( )

( )( )⎟⎠⎞

⎜⎝⎛ ++×

×+−

θαθω

θω

cos1sin2

sinc

cos11

cb

ebd

ci n

Leskova, T. A., Maradudin, A. A. and Zierau, W.(2006)'Synthetic spectra from rough surface scattering',Waves in Random and Complex Media,16:4,531 — 544

Heinz-Hermann Adam

Results

• Performance – 1 typical run•Pentium 4: 9.5 hours•Core 2 Duo: 6.4 hours•Itanium² Montecito: 4 hours

• Elapsed time with 4 CPUs: 1 hour

• Conclusion•Larger Matrices are

possible• > 10,000 x 10,000 for a more

realistic

•64bit are necessary forthese size of matrices

11.05.200911

Heinz-Hermann Adam

• A lot of fruitful scientific calculations are being madepossible in the science departments at the University of Münster.

• Easy access to the Superdome for both batch andinteractive use makes it possible, that even lessexperienced graduate and undergraduate studentscan perform calculations efficiently and concentrateon the science problems.

11.05.2009

Conclusion

13