© 2011 IBM Corporation New HPC Platforms in Academic Research Dr. Theodor Borangiu University Relations Coordinator IBM Romania

© 2011 IBM Corporation

New HPC Platforms in Academic Research

Dr. Theodor BorangiuUniversity Relations Coordinator IBM Romania

2

University Relations – IBM Romania

Global UR Call May 19, 2011 © 2011 IBM Corporation

2011 Highlights in HPC for Academia

1. The GEEA Project: Centre of GRID multi-core high performance resources for research, technological development and innovation support at EU level

2. Blue Gene / P: Supercomputer at the West University of Timisoara

3. UR Investment in academic research lines: Shared IBM-University Research (UPB, WUT)

4. Student research: involving PhD students in HPC research at IBM

5. UR Investment model at UPB and WUT: curricula, skills and academic research with IBM

3



1. The GEEA Project

1. Project description: GEEA – “Centre of GRID multi-corE high pErformAnce resources for research, technological development and innovation support at EU level”

– Owner: University Politehnica of Bucharest (http://www.upb.ro ) - UPB

– Host: CS Department, Faculty of Automatic Control & Computer Science

– Project Director: Professor Nicolae Tapus, PhD, Vice Rector of UPB

– Funding: EU, Operational Sector Program Growth in Economic Competitiveness, POSCCE

– TCV for IBM: 400 K$US

– Inauguration: April 14, 2010

4



1. The GEEA Project

2. Project objectives: GEEA will provide growth of the national R&D capacity of academic and research institutions in strategic domains, and competitiveness for their participation in international projects.

– The GRID Centre was created to provide support for scientific research, technological development and innovation in large scale, distributed computing architectures.

– Development of the national GRID system: development and efficiently interconnecting existing GRID resources [the National GRID Initiative – N_GRID_IN: 7 R&D platforms at country level]; improves services at network level; solutions for accessibility, utilization, efficient control, monitoring, fault-tolerance and security; development of HPC complex applications.

– The training component of the platform – 3 domains: Configuring the high performance GRID communication infrastructure; Using and developing middleware; Development of innovative GRID applications.

– The GEEA Centre hosts the largest GRID platform of the N_GRID_IN: developed during 2009-2011, 7 institutions, IBM TCV = 1.4 M$US

5



1. The GEEA Project

3. GEEA Architecture:

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter H

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter H

DS3524

8

9

16

17

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

DS3524

8

9

16

17

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

DS3524

8

9

16

17

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

300GbFC

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter H

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter H

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

HS21

CD

QS22

CD

QS22

CD

QS22

CD

QS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

BladeCenter H

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

CD

LS22

8 Gbps FC

dual Xeon Nehalem

dual quad-Xeon

Power5& others

dual six-core Opteron

Infiniband

LustreFS over Infiniband

FC Disk array Other Hardware

10 Gbps

UPB RoEDU

Equipment Total From the GEEA project

No. of installed cores/processors 780 168

TerraByte storage solution 25 TB 11 TB

Equipment for infrastructure 80 routers, 48 switches 80 routers, 28 switches

Equipment for network security 11 ASA, 6 IPS-uri 11 ASA, 6 IPS-uri

Infrastructure access 12 stations 12 stations

Heterogeneous structure for intensive computational applications.

Systems partition:

• HPC production

• Preproduction & training

• Virtualization

Available architecture:

• 32 HS21 Dual Intel Quad-Core Xeon E5504 @2.00GHz

• 20 LS22 Dual Opteron Six-Core AMD Processor 2435 @2.6GHz systems

• 4 QS22 Dual IBM PowerXCell 8i @3.2GHz systems

• 4 Sisteme HS22 Dual Intel Quad-Core Xeon E5630 @2.5GHz systems

6



1. The GEEA Project

3. GEEA Architecture (cont.):

At present, users can run up to 420 virtual machines in parallel, accessing up to 20 different network vlans and with unlimited access at all monitoring tools. The data centre disposes of machines dedicated to cloud infrastructure based on Eucalyptus and access to IBM DACS middleware to extend the hybrid programming paradigm for the cluster servers equipped with AMD processors.

Storage:

• Filesystem NFS of 1.8TB

• Filesystem Lustre of 11TB

• Storage space for virtualized systems: 15TB

Interconnection solutions (for storage and data processing modules):

(1) QDR 4x Infiniband

(2) 10 Gigabit Ethernet

(3) Dual Gigabit Ethernet

(4) Fibre Channel

7



1. The GEEA Project

4. Scientific activities and partnership:

Research subjects (currently)– Computational Fluid Dynamics (OpenFOAM, Code Saturne); Material Sciences (Gaussian09, GULP,

GAMESS, Gromacs); Meteorology (HRM, COSMO, WRF); Mathematical Modelling (MathCad, Matlab); Astrophysics (Gadget-2, Celestia); Scientific Visualization (Paraview, ROOT); Interactive Body Physics (Bullet Physics)

International projects with GEEA– FP7: P2P – Peer-to-Peer Next

– FP7: SENSEI – Integrating the Physical with the Digital World of the Network of the Future

– FP7: EUWB – Coexisting Short Range Radio by Advanced Ultra-Wideband Radio Technology

– FP7: TWISNet – Trustworthy Wireless Industrial Sensor NETworks

National partnership with GEEA – ICI – National Institute of R&D in Informatics

– INCAS – National Institute of Aerospace Research “Elie Carafoli”

– IFIN – National Institute of Physics and Nuclear Engineering “Horia Hulubei”

– INFP – National Institute of Earth Physics

– Universities: University of Bucharest, Technical University of Cluj-Napoca; West University of Timisoara, University Alexandru Ioan Cuza of Iasi

Users of the GRID Centre– ICF – Institute of Chemistry “Ilie Murgulescu” of the Academy; ULB Sibiu – University “Lucian Blaga” of

Sibiu; ANM – National Administration of Meteorology; IA – Astronomy Institute of the Academy

8



2. Blue Gene / P

1. Project description: Blue Gene / P Supercomputer at the West University of Timisoara – part of the ICAM infrastructure

– Owner: West University of Timisoara (http://www.uvt.ro ) - WUT

– Host: Faculty of Mathematics and Computer Science

– Project Director: Professor Viorel Negru, PhD, Vice Dean

– Funding: EU, Operational Sector Program Economic Competitiveness Growth – Institute for Advanced Research for the Environment (ICAM)

– TCV for IBM: 1 M$US

– Inauguration: April 20, 2010

9



2. Blue Gene / P

Project objectives: set-up an Advanced Research Institute for the Environment – ICAM (building, labs equipment’s, software and research teams) with a research infrastructure at international standards. One important part of ICAM infrastructure is the High Performance Computing Lab. based on the Blue Gene / P Supercomputer (HPC - Blue Gene / P), which will:

– integrate important research in existing centres of WUT

– stimulate theoretical and applied research, and technological transfer

– provide interdisciplinary approach (environment sciences: physics, chemistry, CS and mathematics, bioinformatics, environment economics) for complex problems solving

– provide cooperation means and international visibility of research results of WUT members

– contribute to the creation of a regional pole of excellence in the Timisoara area the mission of which is to perform research in environmental problems:

environment monitoring, integrated alert system, improving quality of the environment parameters in the Timisoara town (300 000 inhabitants) – West University of Timisoara

Energy efficiency, energy saving, renewable energy – wind for the Timisoara town – University Politehnica of Timisoara

10



1. Blue Gene / P

3. Supercomputer Architecture: A High Performance and High Scalable Computing solution was considered to sustain the research needs at WUT at present and for the next years. The solution provides more than 11 TFps, is visible in Top500.org and Green500.org. There were formulated requirements of high performance computing, scalability, complex mathematical modelling and parallel computation capabilities for specific environment research applications. Also the site conditions were limited by space, power and cooling facilities.

The IBM supercomputer solution consists of:

• One BlueGene/P rack with 4096 cores and 4TB RAM, 32 I/O nodes and 11,72 Tfps Rmax

• 4 power servers p520

• 2 DS3524 and EXP3000 – totally 2x48 SAS HDD

• One Cisco Nexus 7010 Switch with 4x10GbE and 98x1GbE

• SUSE Linux for Power and BG/P specific software

The solution is interconnected with the all customer IT services and infrastructure of WUT.

11



3. UR investment in academic research lines (UPB, WUT)

EquiPoly : Creating a modern R&D infrastructure for advanced Grid computing and a testbed for various technologies – an IBM Equinox 2005 awarded R&D project of the CS Dept. of University Politehnica of Bucharest [UPB] (Professor Nicolae Tapus Director)

The Equinox program identifies student and research communities starting leading-edge visible IT projects and supports these projects. UPB receives an IBM grant of xSeries entry-servers to run IBM software for Linux. EquiPoly infrastructure used to develop research in GRID computing with the targets:

– develop a cluster based on xSeries and LINUX

– middleware software for cluster

– optimization algorithms for complex distributed systems

– distributed dynamic network services and development of collaborative applications

12



3. UR investment in academic research lines Collaborative Research, Awards, Research stages and Road showsCollaborative Research, Awards, Research stages and Road shows

– IBM Faculty Awards 2008:– Cell-Based Processing of Video and Medical Data Flows from Ambulance NetworkCell-Based Processing of Video and Medical Data Flows from Ambulance Network

(Emergmed_Cell), Professor Dan Popescu, UP Bucharest– Cell GAF – A Genetic Algorithms Framework for the Cell Broadband EngineCell GAF – A Genetic Algorithms Framework for the Cell Broadband Engine, Associate

Professor Emil Slusanschi, PhD, UP Bucharest

– IBM Real-Time Innovation Awards 2008: – Verifying Real-Time Programs with Separation LogicVerifying Real-Time Programs with Separation Logic, Professor Nicolae Tapus, UP Bucharest

– IBM Open Collaborative Research 2009:– CEEMEA Blue Gene Research Collaboration and Community BuildingCEEMEA Blue Gene Research Collaboration and Community Building (with Moscow State

University, the Nile University, the Astronomy Institute, the National Authority for Meteorology

– IBM Country Projects 2011:

– Smarter Black SeaSmarter Black Sea, a KEE for the Danube, Delta and Black Sea ecosystem, HPC with Blue Gene in collaboration: West University of Timisoara and University Ovidius of Constanta

– Research Stages:– Emil Slusanschi, Alex Herisanu from UPB at IBM Watson Res. Lab. for Advanced CELL

Programming, Cell/BE applications development in image processingCell/BE applications development in image processing

– Cell BE Programming 1 and 2 road shows: – 4 road shows for student training in Cell BE programming4 road shows for student training in Cell BE programming at the technical universities of:

Timisoara, Cluj-Napoca, Iasi, Suceava, trainers: Emil Slusanschi, Alex Herisanu / CS Dept. of UPB

Collaborative Research, Awards, Research stages and Road showsCollaborative Research, Awards, Research stages and Road shows– IBM Faculty Awards 2008:

– Cell-Based Processing of Video and Medical Data Flows from Ambulance NetworkCell-Based Processing of Video and Medical Data Flows from Ambulance Network (Emergmed_Cell), Professor Dan Popescu, UP Bucharest

– Cell GAF – A Genetic Algorithms Framework for the Cell Broadband EngineCell GAF – A Genetic Algorithms Framework for the Cell Broadband Engine, Associate Professor Emil Slusanschi, PhD, UP Bucharest

– IBM Real-Time Innovation Awards 2008: – Verifying Real-Time Programs with Separation LogicVerifying Real-Time Programs with Separation Logic, Professor Nicolae Tapus, UP Bucharest

– IBM Open Collaborative Research 2009:– CEEMEA Blue Gene Research Collaboration and Community BuildingCEEMEA Blue Gene Research Collaboration and Community Building (with Moscow State

University, the Nile University, the Astronomy Institute, the National Authority for Meteorology

– IBM Country Projects 2011:

– Smarter Black SeaSmarter Black Sea, a KEE for the Danube, Delta and Black Sea ecosystem, HPC with Blue Gene in collaboration: West University of Timisoara and University Ovidius of Constanta

– Research Stages:– Emil Slusanschi, Alex Herisanu from UPB at IBM Watson Res. Lab. for Advanced CELL

Programming, Cell/BE applications development in image processingCell/BE applications development in image processing

– Cell BE Programming 1 and 2 road shows: – 4 road shows for student training in Cell BE programming4 road shows for student training in Cell BE programming at the technical universities of:

Timisoara, Cluj-Napoca, Iasi, Suceava, trainers: Emil Slusanschi, Alex Herisanu / CS Dept. of UPB

13



4. Involving Ph.D. students in HPC research at IBM PhD Fellowship programPhD Fellowship program:

– 2006-2007: Efficient Task Scheduling in Grid Environments Efficient Task Scheduling in Grid Environments, Corina Stratan, PhD student 3rd year in CS Dept. of PU Bucharest [thesis director Theodor Borangiu]

– Thesis ObjectivesThesis Objectives: design efficient strategies for job scheduling in large scale Grid environments, which aim to obtain an optimal balance between execution time and cost

– Inter-cluster scheduling involving simultaneous management of multiple clusters. Meta-scheduling solution developed: de-centralized, based on agents, highly scalable and fault tolerant. Additional contribution: evaluation of job execution costs

– 3-month Internship3-month Internship (2006) at IBM T.J. Watson Research Center, Dept. of Distributed Systems and Resource Management, Award renewed in 2007-2008Award renewed in 2007-2008

– 2006-2007: Optimization of De-centralized Scheduling Strategies in Grid Environments,Optimization of De-centralized Scheduling Strategies in Grid Environments, Florin Pop, PhD student 2rd year in CS Dept. of PU Bucharest [thesis director Valentin Cristea]

– Thesis ObjectiveThesis Objective: determine optimal strategies for application scheduling in large scale Grid environments. An approach based on genetic algorithms is used.

– Proposing a solution for decentralized scheduling; Obtaining an optimal load balancing between execution time and cost; Defining an evaluation model for scheduling strategies

– 2008-2009: TTechniques for the Optimization of Communication Flows in Distributed Systemsechniques for the Optimization of Communication Flows in Distributed Systems ”, ”, Mugurel Andreica, PU Bucharest [thesis director Nicolae Tapus]Mugurel Andreica, PU Bucharest [thesis director Nicolae Tapus]

– 2009/2010: Data Storage, Representation and Interpretation in Grid Monitoring EnvironmentsData Storage, Representation and Interpretation in Grid Monitoring Environments, Alexandru Costan, UP Bucharest [thesis director Valentin Cristea]

– 2010/2011: Applying Emergence and Self -Organization to Engineer High Quality Autonomic Applying Emergence and Self -Organization to Engineer High Quality Autonomic SystemsSystems, Bogdan Caprarescu, West Uni. of Timisoara) [thesis director Viorel Negru]

Two-mentor directing teamTwo-mentor directing team: IBMer (UR, SWG, STG) and University Professor: IBMer (UR, SWG, STG) and University Professor

14



4. UR investment model at UPB and WUT Curricula

– Annual analysis of faculty curricula (engineering, economics, fundamental sciences)

– IBM exec. staff – members of Advisory Boards in Faculty Professor Councils (UPB, WUT); regular meetings

– IBM takes active part in education plan design, curricula update

Skills– Identify faculty needs in conjunction with current research strategy

– Map IBM technologies on current faculty needs

– Organize trainings (TTT). Frameworks: Country Projects, direct funding by local organization, involving BPs, utilization relief (SWG, STG, GBS), using campus programs (Cell B/E, Cloud, Grid)

– Stimulate professors to write down the course with IBM support for hands-on labs and exercises

Preparing Research – Update professor knowledge about HPC equipment and technologies (Academic Days event,

technical days at universities (WUT, UPB), direct meetings with academic staff)

– Use University Partnership Executive Program: CGM and IBM execs. meet Rector and University execs. in H1 and H2 of each year; R&D and strategy plans of university are discussed

– University asks IBM to provide technical support during submission of HPC-based applications

Assisting University Research– Collaboration in FA, SUR, OCR (technical briefings, documentation), Ph.D support; consultancy

provided; use STG University Alliance facilities

– IBM facilitates international partnership

15



Thank you !

Documents

© 2011 IBM Corporation New HPC Platforms in Academic Research Dr. Theodor Borangiu University Relations Coordinator IBM Romania