NATIONAL SUPERCOMPUTING CENTRE SINGAPORE · NATIONAL SUPERCOMPUTING CENTRE SINGAPORE . ... Singapore • State-of-the-art national facility with computing, data and ... (GPU Center

PROF. TAN TIN WEE, DIRECTOR, NSCC SINGAPORE

JON LAU, DEPUTY DIRECTOR, NSCC SINGAPORE

DR. GABRIEL NOAJE, SENIOR COMPUTATIONAL SCIENTIST, A*STAR CRC

NATIONAL SUPERCOMPUTING

CENTRE SINGAPORE

National Supercomputing Centre Singapore

• State-of-the-art national facility with computing, data and

resources to enable users to solve science and technological

problems, and stimulate industry to use computing for

problem solving, testing designs and advancing technologies.

• Facility will be linked by high bandwidth networks to connect

these resources and provide high speed access to users anywhere

and everyone.

National Supercomputing Centre (NSCC) Singapore

August 27, 2015 3 GTC Asia South 2015


Supporting National R&D Initiatives 1

Objectives

Attracting Industrial Research Collaborations 2

Enhancing Singapore’s Research Capabilities 3



Petascale 2015 National Supercomputing Centre (NSCC)

2006 A*STAR Computational Resource Centre (ACRC)

Mid 2000s HPCC of Nanyang Technological University (NTU)

1998 Institute for High Performance Computing (IHPC)

Early 1990s National Supercomputing Resource Centre

(NSRC)

Late 1980s DISCS and Computer Centre, National University

of Singapore (NUS)

Historical Trajectory


Terascale

Gigascale


NSCC Datacentre - Fusionopolis

8

Level 17

August 27, 2015 GTC Asia South 2015


NSCC Datacentre



Pillars of NSCC

1 PF

supercomputer

HPC

domain

experts

STAR-N

network



NSCC components

NSCC Supercomputer • New 1+ PetaFLOP Supercomputer

• Recurrent investment every 3-5 years

• Co-investment (OpEx) from primary stakeholders

• Remote login nodes in stakeholders campuses

Science, Technology & Research

Network (STAR-N)

• High bandwidth InfiniBand fabric to connect the distributed login nodes

• High speed network access to users (both public & private) anywhere

• Support transfer of large data-sets (both locally & internationally)

• Build local & international network connectivity (Internet 2 USA, TEIN*CC to

GEANT, to Japanese Networks)

• ASEAN, USA, Europe, Australia, Japan, Middle East



Funding (MTI)

& Co-funding

(A*STAR, NUS,

NTU)

Approved

November

2014

Tender Open 20 January 2015

Tender Closed 14 April 2015

Tender Awarded 15 June 12015

Facility ETA Q4 2015

Tender Process



NSCC Supercomputer

• Workflow pipeline (from sequencer to memory/storage) + interactive access built in • Just-in-time resource delivery i.e. interactive access for some workloads (genomics) • Warm water cooled

~1 PFlop

System

• 1,288 nodes (dual socket,

12 cores/CPU E5-2690v3)

• 128 GB DDR4 / node

• 10 Large memory

nodes (1x6TB, 4x2TB, 5x

1TB)

~10PB Storage

• HSM, Tiered, Tier 3

SSD disks

• I/O 500 Gbps flash

burst buffer , 10x

Infinite Memory

Engine (IME)

EDR

Interconnect

• Mellanox EDR Fat Tree

• 500 Gbps pipeline

between Biopolis &

Fusionopolis

• InfiniBand connection

to all end-points (login

nodes) at three campuses



GPU nodes

Accelerators nodes

• 128 nodes with NVIDIA GPUs (identical to the compute nodes)

• NVIDIA K40 (2880 cores)

• 368,640 total GPU cores

Visualization nodes

• 2 nodes Fujitsu Celsius R940 graphic workstations

• Each with 3 x NVIDIA Quadro K4200

• NVIDIA Quadro Sync support



Singapore InfiniBand Connectivity

Connecting all National Supercomputing Centre stakeholders:

A*STAR, NUS, NTU, SUTD and others with InfiniBand links.

NUS

A*STAR

Fusionopolis

SingAREN

Global Switch

100Gbps InfiniBand enabled

500Gbps InfiniBand enabled

A*STAR

Biopolis

WOODLANDS

10/40/100Gbps InfiniBand/IP

SELETAR

CHANGI

NOVENA

OUTRAM

ONE-NORTH

JURONG

NTU



Example of Accessibility NSCC-GIS Integration - Big Data with HPC

August 27, 2015 GTC Asia South 2015 16

NGSP Sequencers at B2

(Illumina + PacBio)

NSCC

Gateway

GIS

NSCC

STEP 2: Automated pipeline analysis once

sequencing completes. Processed data

resides in NSCC

500Gbps

Primary link

Data Manager

STEP 3: Data manager index and

annotates processed data. Replicate

metadata to GIS. Allowing data to be

search and retrieved from GIS

Data

Manager

Compute Tiered Storage

POLARIS, Genotyping

and other Platforms in

L4~L8

Tiered Storage

STEP 1: Sequencers stream

directly to NSCC Storage (NO

footprint in GIS)

Compute

1 Gbps per sequencer

10 Gbps

1 Gbps per machine

100 Gbps

10 Gbps


Global Connectivity as Anchor Participant of InfiniCortex Project


NSCC


HPC Expertise

• Engineers available to help users initiate, optimize, migrate

their applications on the new supercomputer

• Support team available for hardware and software issues

through a ticketing system

• Partnerships with experts like NVIDIA, Mellanox,

Obsidian, DDN, etc.

• Collaborative projects with international academic and

research supercomputer centres



GPU Outreach Efforts

• NSCC’s system will have 128 GPU accelerated nodes as a start

• Capacity will be increased once the demand is high enough

• GPU Computing is still quite a new domain in Singapore

• Exploring opportunities to develop competency of Parallel

Computing and GPU acceleration locally and in the region

• Provide HPC training sessions as well as advanced courses (GPU

programming, code optimization)

• Exploring collaboration projects to use the 128 GPUs



GARUDA from Systems Biology Institute

Garuda – The way biology connects

• open, community-driven, common platform that provides a

framework to interface, discover, & navigate through different

applications, databases and services in bio-medical research

• provides language independent API to connect software as

gadgets

• Gadget = software integrated as a node in a more complex

workflow

• explore gadgets through the gateway and operate them

through the dashboard

• supported by a global alliance of leaders in computational

biology and informatics



GARUDA on GPUs

• Extend the gadget portfolio to Deep Learning applications

• Install the entire framework on the NSCC supercomputer to

allow for a low entry point into HPC for researchers


• Integrate GPU-enabled

bioinformatics applications as

gadgets into the Garuda

framework

• Provide Garuda users with

access to workflows that

include GPU-enabled

applications


VASP

• VASP = Vienna Ab-initio Simulation Package

• a complex package for performing ab initio quantum-

mechanical molecular dynamics (MD) simulations using

pseudopotentials or the projector-augmented wave method

and a plane wave basis set

• VASP+GPU version 5.4.1 developed by Univ of Vienna with

newer collaborators (ENS-Lyon, IFPEN, U of Chicago, CMU,

URCA, Materials Design, Inc., NVIDIA and Aachen-RTWH)

• A*CRC / NSCC part of the closed beta test program

• Up to 3x speedup using 4 x K40 in a single node compared to

CPU only



GROMACS

• GROMACS = GROningen MAchine for Chemical Simulations

• a versatile package to perform molecular dynamics, i.e.

simulate the Newtonian equations of motion for systems with

hundreds to millions of particles

• version 5.1 released on August 14, 2015 with many GPU

improvements

• Improvements to mdrun performance

– Improved performance auto-tuning with GPUs

– Added kernel and compiler support for K80

– Added optional NVIDIA Management Library (NVML) Integration

– Allowed increasing CUDA thread block size

– Added support for flushing WDDM queue on Windows

– Optimized atomic accumulation in CUDA short-ranged kernels

– Improved the intra-GPU load balancing



Unit A of human estrogen receptor A

• 78.000 atoms

• Explicit solvent

• OPLS-aa wer-a-a-bpa-ion-

pull.groforce field

• Gromacs 5.0.6 with CUDA 7.0

Performance

• CPU-only run: 56 ns/day

• With 1x K40m: 85 ns/day

• With 2x K40m: 130 ns/day

• With 1x K80: 120 ns/day

• With 2x K80: 146 ns/day

Challenges

• Finding the best ration between

number of OpenMPI/OpenMP

tasks

GROMACS



Blender

• Collaboration with digital media SME to fit open source

Blender into animation pipeline

• Leveraging on Blender optimized on NVIDIA GPUs

• Benchmarking of Blender with and without GPUs

• POC one of the largest Blender rendering farms


http://www.blender.org/wp-content/uploads/2012/11/Modelling1.png


Demo and prototyping novel services

InfiniCloud-GPU

• Rapid scheduling of GPU accelerated cloud instances on

InfiniCloud – InfiniBand interconnected cluster

• GPU passthrough support at the KVM

Altair PBS Pro & Bright Computing BCM • Scheduling of accelerated cloud instances on High

Memory Machines (12 TB RAM)

• Scheduling of accelerated cloud instances on GPU nodes



GPU Meetups Around the World



Singapore GPU Users Group


Singapore GPU Users Group in numbers:

• 2 organizers (Dr. Gabriel Noaje - A*CRC & Oxana

Plis – NVIDIA)

• 10 meetups since October 17th, 2013

• 20+ speakers

• 213 registered members, ~30 participants/meeting

• Giveaways: GPU, Shield tablet, NVIDIA Shield


HPC democratization Co-organize/sponsor local events:

• GPU Technology Conference Asia South

• Deep Learning Workshop

Trainings:

• Comprehensive training program in both HPC in general and

GPU computing in particular

• Collaborate with partners: University of Illinois Urbana-

Champagne (GPU Center of Excellence), IDA, NVIDIA

• Different levels: beginners, advanced, topic oriented

• Different platforms: classic face to face courses, self-paced

online courses, interactive sessions

• Challenges for students, young entrepreneurs to promote and

incentivize HPC usage across Singapore and SE Asia

• Many initiatives to come …



Conclusions

• NSCC = National Supercomputing Centre

• 1 PF Supercomputer (not very big, but unique in many

ways)

• Objective: democratize HPC in all research and industry

domains across Singapore

• Create an HPC ecosystem in the region through:

–Aggressive promotion and partnership development strategy

–High speed interconnect with the world and local sites

islandwide

– Lead GPU centered projects and initiatives



We want to work with you !



THANK YOU

Documents

NATIONAL SUPERCOMPUTING CENTRE SINGAPORE · NATIONAL SUPERCOMPUTING CENTRE SINGAPORE . ... Singapore • State-of-the-art national facility with computing, data and ... (GPU Center