Stockholm Brain Institute Blue Gene/L...4 October 18, 2007 Blue Gene Technology Roadmap 1GB Version Available 1Q06 Blue Gene On Demand Center Blue Gene/Q (Power architecture) Targeting

IBM® System Blue Gene®/P Solution © 2007 IBM Corporation1

Stockholm Brain Institute Blue Gene/L


Stockholm Brain Institute Blue Gene/L

© 2002 IBM Corporation

IBM Systems & Technology Group and IBM Research

Copyright 2007– IBM Corporation

IBM® Blue Gene®/P - An Overview of a PetaflopCapable System

Carl G. TengwallIBM Storage & Technology [email protected]

4 October 18, 2007

Blue Gene Technology Roadmap

1GB VersionAvailable 1Q06

Blue GeneOn Demand

Center

Blue Gene/Q(Power architecture)Targeting 10 PFlops

Perf

orm

ance

2004 2011

Blue Gene/P(PPC 450 @ 850MHz)Scalable to 3 PFlops

2007

Blue Gene/L(PPC 440 @ 700MHz)

Scalable to 360 TFlops

Top500 list June 2007


Blue Gene/P Architectural Highlights


Blue Gene/P Architectural HighlightsLightweight kernel (CNK) on Compute NodesLinux on I/O Nodes handling syscallsOptimized MPI library for high speed messagingControl system on Service Node with private control networkCompilers and job launch on Front End Nodes


Blue Gene/P compared to Blue Gene/L


13.6 GF/s8 MB EDRAM

4 processors

1 chip, 20 DRAMs

13.6 GF/s2.0 (or 4.0) GB DDRSupports 4-way SMP

32 Node Cards1024 chips, 4096 procs

14 TF/s2 TB

72 Racks

1 PF/s144 TB

Cabled 8x8x16Rack

System

Compute Card

Chip

435 GF/s64 GB

(32 chips 4x4x2)32 compute, 0-1 IO cards

Node Card

Offering Schematic – Blue Gene/P

Front End Node / Service NodeJS21 / Power5Linux SLES10

Blue Gene/P continues Blue Gene’s leadership performance in a space-saving, power-efficient package for the most demanding and scalable

high-performance computing applications

HPC SW:CompilersGPFSESSLLoadleveler


Blue Gene/P Interconnection Networks3 Dimensional Torus

– Interconnects all compute nodes (73,728)– Virtual cut-through hardware routing– 3.4 Gb/s on all 12 node links (5.1 GB/s per node)– 0.5 µs latency between nearest neighbors, 5 µs to the farthest– MPI: 3 µs latency for one hop, 10 µs to the farthest– Communications backbone for computations– 1.7/3.9 TB/s bisection bandwidth, 188TB/s total bandwidth

Collective Network– One-to-all broadcast functionality– Reduction operations functionality– 6.8 Gb/s of bandwidth per link– Latency of one way tree traversal 1.3 µs, MPI 5 µs – ~62TB/s total binary tree bandwidth (72k machine)– Interconnects all compute and I/O nodes (1152)

Low Latency Global Barrier and Interrupt– Latency of one way to reach all 72K nodes 0.65 µs,

MPI 1.6 µs

Other networks– 10Gb Functional Ethernet– I/O nodes only– 1Gb Private Control Ethernet– Provides JTAG access to hardware. Accessible only from Service

Node system


Blue Gene/P System in a Complete Configuration

Complete Blue Gene/P Solution

BG/Pcompute

nodes1024 per Rack

BG/PI/O nodes

8 to 64per Rack

Fede

rate

d 10

Gig

abit

Eth

erne

t Sw

itch

Front-end nodes

Service node

WAN

visualization

archive

FS

1Gb/s - 4 per Rack

Control network

10Gb/s per I/ONode

1 to 7

2 Rac

ks

Storage Nodes File Storage


Blue Gene/P has wider rack dimensions and slightly longer vents than Blue Gene/L, but is otherwise similar.


BPC ASIC

4 cores, 8MB

Cu Heatsink

SDRAM-DDR2

1 of 20 sites.

DRAM address termination (Vtt),

EEPROM, decoupling, monitoring

13

PPC 450

FPUL1 PrefetchingL2

4MBeDRAM

L3

DDR-2Controller

PPC 450

FPUL1 PrefetchingL2

PPC 450

FPUL1 PrefetchingL2

4MBeDRAM

L3

DDR-2Controller

PPC 450

FPUL1 PrefetchingL2

Torus Collective BarrierJTAG 10Gb Ethernet

DMA

Blue Gene/P node

Arb

6*3.4Gb/sbidirectional



ControlNetwork

2*16B bus @435Mb/s

To 10Gbphysical layer

(Shares I/O with Torus)

Multiplexing sw

itchM

ultiplexing switch

13.6GB/s read(each), 13.6GB/s write(each)

Data read @ 6.8GB/sData write @ 6.8GB/sInstruction @ 6.8GB/s

13.6GB/s external DDR2 DRAM bus

4 symmetric ports for Tree, torus and global barriers

DMA module allowsRemote direct “put”/ “get”

14

Main Memory Bandwidth per Rack

0

2000

4000

6000

8000

10000

12000

14000

NASAItanium

Cray X1E

Cray XT4

ASCIPurple

BG/P NEC SX-8

15 October 18, 2007

0.230

0.350

0.0290.020 0.022 0.018

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

BG/L BG/P OpteronCluster

Cray RedStorm

Thunderbirddual-Xeon

Purple

System Power Efficiencygi

gaFL

OP/

s pe

r Wat

t


Blue Gene Software Hierarchical OrganizationCompute nodes dedicated to running user application, and almost nothing else - simple compute node kernel (CNK)I/O nodes run Linux and provide a more complete range of OS services – files, sockets, process launch, signaling, debugging, and terminationService node performs system management services (e.g., heart beating, monitoring errors) - transparent to application software


Supporting Equipment

Front End Nodes– IBM system-p 64-bit servers (not Blue Gene)

– Used for development of application code

– XL compilers, GNU compilers, libraries availableStorage Nodes

– A parallel filesystem is generally shared between to Blue Gene (the I/O nodes) and the Front End Nodes


Blue Gene SoftwareCompilersMessage Passing LibraryESSL & MASS LibrariesGPFS File SystemLoadLever schedulerHPC ToolkitManagement software on Management NodeCompute Node Kernel on Compute NodesLinux on I/O nodes


Materials Science

Climate Modeling

Genome Sequencing BiologicalModeling

Pandemic Research

Fluid Dynamics

Drug Discovery

Geophysical Data Processing

What is Blue Gene used for?

Financial Modeling


Applications already tested and running on BG/PBenchmarks

– Linpack, HPC Challenge– NAS Serial, NAS OpenMP, NAS Parallel– PALLAS, STREAM, MPPTest– UMT2K, SPPM, SPHOT

Applications– Physics: SPHOT, QCD, MILC, ICEPIC– Weather/Climate: HOMME, HYCOM, WRF, POP– Astrophysics: FLASH– CFD: Raptor, AVUS, CTH, AMR– Molecular Dynamics: NAMD, LAMMPS– Quantum Chemistry: CPMD, GAMESS– Materials Science: ParaTEC

All applications from BG/L will run on BG/P after recompile– General performance improvement of 2.4 going from BG/L to BG/P– Some applications have ratio >2.4 due to “superscaling”

• SMP mode with 4 threads on BG/P can achieve better performance than BG/L nodes

Many more applications in progress of being ported over and tested


Linpack on Blue Gene/P

ISC2007 Top 500 submission– 2 Racks, 1 MPI Process / Core, 8192 MPI Processes Total

• used basic NETLIB HPL with tweaks• used BG/L version of ESSL

– 20.86 TF, 74.89% of peak

– More performance to come


Scalability – Continues Extraordinary Blue Gene Behavior

LAMMPS Scalability

0

1

2

3

4

5

6

7

8

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

CPU Count

Rela

tive

Perfo

rman

ce

HYCOM Scalability

0

1

2

3

4

5

6

7

0 500 1000 1500 2000 2500 3000 3500 4000 4500

CPU Count

Rel

ativ

e P

erfo

rman

ce

Two Examples– LAMMPS - molecular dynamics

– HYCOM - ocean modeling


Blue Gene is addressing six critical issues on the path to Petaflop computing

1. Power2. Floor space3. Cost4. Single processor performance5. Network scalability6. Reliability

Summary


Thank you!


Source: www.top500.org


IBM supercomputing leadership

Source: www.top500.org

Semiannual independent ranking of top 500

supercomputers in the world

IBM leads several key TOP500 metrics ...#1 System – LLNL – Blue Gene/L (280.6 TF)Most installed aggregate throughput with over 2,060 Teraflops (41.7%)Most in TOP10 with 6 systems (40%)Most in TOP100 systems with 45 (45%)Fastest machines in USA (BG/L)Fastest machine in Europe (MareNostrum)Fastest machine in China (Sinopec)

June 2007 Aggregate PerformanceIBM: 2,060 of 4,946 TF (41.9%)

IBM41,7%

SGI5,7%

HP24,2%

Other7,2%Dell

8,8%

Sun0,9%

Cray7,3%

Linux Networx

2,1%

NEC2,1%

June 2007 TOP100IBM is clear leader with 46%

IBM; 45

HP; 7 Dell; 13

Apple; 2

SGI; 9

Cray; 9

Other; 11Linux

Networx; 4


More Information

IBM Redbooks for Blue Gene ibm.com/redbooks

– Application Development Guide

– System Administration Guide

– Performance ToolsOpen Source Communities


Read more about it…

http://www.ibm.com/servers/deepcomputing/bluegene.html

Sign up for newsletter

http://www-fp.mcs.anl.gov/bgconsortium/default.htm

Sign up for a consortium membership

http://www.top500.org

Latest TOP500 list

http://www.ibm.com/servers/deepcomputing/bluegene.htmlhttp://www-fp.mcs.anl.gov/bgconsortium/default.htmhttp://www.top500.org/

IBM® Blue Gene®/P - An Overview of a Petaflop Capable System Blue Gene Technology RoadmapBlue Gene/P Architectural HighlightsBlue Gene/P Architectural HighlightsBlue Gene/P compared to Blue Gene/LOffering Schematic – Blue Gene/PBlue Gene/P Interconnection NetworksBlue Gene/P System in a Complete ConfigurationBlue Gene/P has wider rack dimensions and slightly longer vents � than Blue Gene/L, but is otherwise similar. Main Memory Bandwidth per RackBlue Gene Software Hierarchical OrganizationSupporting EquipmentBlue Gene SoftwareApplications already tested and running on BG/PLinpack on Blue Gene/PScalability – Continues Extraordinary Blue Gene BehaviorBlue Gene is addressing six critical issues on the path to Petaflop computingThank you!More InformationRead more about it…

Documents

Stockholm Brain Institute Blue Gene/L...4 October 18, 2007 Blue Gene Technology Roadmap 1GB Version Available 1Q06 Blue Gene On Demand Center Blue Gene/Q (Power architecture) Targeting