Challenge in Building Challenge in Building National Scale Grid National Scale Grid InfrastructureInfrastructure

Somsak SriprayoonsakulSomsak SriprayoonsakulThai National Grid CenterThai National Grid CenterSoftware Industry Promotion Agency (SIPA)Software Industry Promotion Agency (SIPA)Ministry of Information and CommunicationMinistry of Information and Communication

Thai National Grid ProjectThai National Grid Project• A 4 years project

under Ministry of Information and Communication Technology

• 5.5 Million US$ program

• Start from 13 universities and 1 government organizations

• Extending rapidly

MinisterMICT

National Grid Committee

DirectorSIPA Board

SIPA

AdminStaffs

Member Institutes

DirectorThai National Grid Center

Researchers

BoardThai National Grid Center Associate Director

Thai National Grid Center

Partner InstitutesPartner Institutes

Bangkok

Chaingmai University (CMU)

Chulalongkorn University (CU)Kasetsart University (KU)King Mongkut’s Institute of Technology North Bangkok (KMITNB)King Mongkut’s University of Technology Thonburi (KMUTT)King Mongkut’s Institute of Technology Ladkrabang (KMITL)Mahidol Unversity (MU)Silpakorn University (SU)Meteorological Department of Thailand (TMD)

Walailak University (WU)

Prince of Songkla University (PSU)

Khonkhan University (KKU)

Suranaree University of Technology (SUT)

Nakhon Ratchasima

Khonkhan

Chaingmai

Nakhonsrithammarat

Songkla

Potential Application Potential Application DomainDomain• Life-science Grid

• Cheminformatics Grid

• Geoinformatics Grid

• Digital Archive and E-learning

• Computational Fluid Dynamics Grid

• Simulation and Modeling Grid

• Digital Media and Animation

2006 2007 2008 2009

Pilot Grid R&E Grid Enterprise Grid

•Pilot Grid and Cluster infrastructure•Interoperable Middleware•Basic computational Grid•Basic Data Grid•POC applications

•Stable Grid and Cluster infrastructure•Interoperable Middleware and basic services•Basic computational Grid•Basic Data Grid•Grid Portal•Grid applications

•R&E •Non production enterprise app

•Broad academic adoption•Limited commercial adoption

•Production Grid and Cluster infrastructure•Rich set of services

•Computational Grid•Enterprise analysis Grid•Data and knowledge Grid•Collaborative Grid

•Broad commercial adoption

Capability

Time

ปั�ตตานี�

ยะลานีราธิ�วาส

แม่�ฮ่�องสอนี

เชี�ยงราย

ล�าปัาง

พะเยา

นี�านี

แพร�

อ�ตรดิ�ตถ์�

ส�โขทั ย

ล�าพ!นี

ตาก

ก�าแพงเพชีร เพชีรบู!รณ์�

เลย

ชี ยภู!ม่�พ�จิ�ตร

หนีองบู วล�าภู!

อ�ดิรธิานี�

หนีองคาย

สกลนีครนีครพนีม่

กาฬส�นีธิ��

ม่หาสารคาม่

ม่�กดิาหาร

ร*อยเอ+ดิ ยโสธิรอ�านีาจิเจิร�ญ

บู�ร�ร ม่ย� ส�ร�นีทัร� ศร�สะเกษ

นีครสวรรค�

อ�ทั ยธิานี�

กาญจินีบู�ร�ส�พรรณ์บู�ร�

ลพบู�ร�

สระบู�ร�

ปัราจิ�นีบู�ร�สระแก*ว

ฉะเชี�งเทัรา

จิ นีทับู�ร�ระยอง

ราชีบู�ร�

เพชีรบู�ร�

ตราดิ

พระนีครศร�

ส�งห�บู�ร �

ปัทั�ม่ธิานี�นีนีทับู�ร �

สม่�ทัรสงคราม่สม่�ทัรสาคร

สม่�ทัรปัราการ

ปัระจิวบูค�ร�ข นีธิ�

ชี�ม่พร

ระนีอง

ส�ราษฏร�ธิานี�

พ งงา

ภู!เก+ต

กระบู�1

ตร ง

สต!ล

พ ทัล�ง

นีครนีายก

เชี�ยงใหม่�

นีครราชีส�ม่าอ�บูลราชีธิานี�

สงขลา

นีครศร�ธิรรม่ราชี

Inter-University Network (UniNet) Inter-University Network (UniNet) Commission on Higher Education (Center)

24 Public Universities

39 Rajabhat Universities29 IT Campus

8 Private Universities13 Mahachulalongkorn Rajawitayalai9 Other agencies9 Rajamangala Institue of Technology (44 Campus) Ministry of Education (for Under Higher Education)

155 Mbps- STM134 Mbps - E38 Mbps - 4*E12 Mbps - E164 k – 2Mbps

CMUCMUSUTSUT

MUAMUA

SU_NMJSU_NMJ

BUBU

PSUPSU

DomesticDomestic1Gbps1Gbps

Internet - 622 MbpsInternet - 622 Mbps

(to Internet 2 155 Mbps)(to Internet 2 155 Mbps)

2.5 Gbps- STM161 Gbps

Update on 24 Jan 2006Update on 24 Jan 2006

THAISARNTHAISARN

1Gbps1Gbps

MOEnet (K-12)MOEnet (K-12)

1Gbps 1Gbps

(coming soon)(coming soon)

Grid InfrastructureGrid Infrastructure

ThaiRen(Uninet/ThaiSarn)

KMITNB

KMUTT

KMITL

TMD

WU

KU

PSU

CMU

SUT

KKU SU

MU

CU

AIT

Teraflop Clusterat TNGC

2.5Gb

NECTEC

ImplementationImplementation

C C C C

H

C C C C

H C C C C

H

C C C C

H

Network

GH Grid Scheduler

Super ClusterGrid Model

Super ClusterGrid Model

Users see grid as a large single cluster

Thai National Grid Thai National Grid Standard Software StackStandard Software Stack

Applications

Core: Gamess , AutodockOption: Blast

Portal: GridSphereOption: GridPort

Management

Core: SCMSOption: MOGAS

Scheduler

Core : SQMS , SGEOption: PBS, Condor, CSF

Programming

Ninf-GMPI -G2

Grid Middleware

Globus Toolkit 4.0

Fabric

Core: Linux / Rocks on x86_64/32 Option: Windows

Network

Building the GridBuilding the Grid

Grid MiddlewareBioinformatics

Digital Media HEP Medical

Technical ContactTechnical Contact

• Each ThaiGrid member institute need to assign at least one person as technical contact point for all ThaiGrid activity

• Technical contact tasks– Software setup (both cluster and Grid)– Contact point for people inside institute

• Our communication medium are mailing-list, webboard, wiki, skype, and telephone– http://rswg.thaigrid.or.th are the central point for all

technical contacts

ThaiGrid deployment stepThaiGrid deployment step

1. Site firewall compliance test

2. Site deployment• Install NPACI Rocks 4.1• Install ThaiGrid Roll

3. Site setup• Grid CA setup• Setup of Grid services• Register site with Grid Control Center node

4. Site Certify

Site firewall compliance Site firewall compliance testtest

• ThaiGrid using Globus 4.0 software that comes with NAPCI Rocks

• Focus on using Pre-WS Grid component – 2119 – Globus Gatekeeper– 2811 – Grid FTP– 2135 – Globus MDS– 80 – For web access– 22 – For SSH access– 10001 – 20000 – reserved for applications

• Firewall is a nightmare for Grid deployment– A Firewall testing tools is needed to identify firewall

problem

Firewall testing toolFirewall testing tool

• A firewall tester program “TGCheckport” is developed.

[bank@tgcc TGcheckport-1.3]$ ./TGcheckport.sh maeka.ku.ac.thStart testing maeka.ku.ac.thTue May 16 16:48:31 ICT 2006determining globus-gatekeeper service [2119] ... Passdetermining gsiftp service [2811] ... Passdetermining globus-mds service [2135] ... Passdetermining scmsweb service [80] ... Passdetermining globus-mpi-required-spare-port service [10001-20000] ... PassCongratulation, you pass all the test. Your system is ThaiGrid FirewallCompliance.Finish testingTue May 16 16:48:32 ICT 2006[bank@tgcc TGcheckport-1.3]$

Site DeploymentSite Deployment

• Site deployment process– OS installation using – Grid core middleware (Globus) installation– Grid additional middleware installation

• MPICH-G2, ninf-G, GSI-ssh

ChallengeChallenge

• All these must be done on every cluster nodes to ensure compatibility for all components

• Manual installation is time consuming, costly, and error prone

• Rocks simplify all of these process due to the roll concept– OS and Middleware installation are highly automatic

ChallengeChallenge

• Many Grid middleware components are needed– SGE/NINF-G/GLOBUS/MPICH/SCE– Each has already been packed into Rocks Roll– Nevertheless, to completely install all software require many

rolls

• Solution: every software are bundled to a new ThaiGrid Roll– A Jumbo Roll bundle all standard Grid software together

ThaiGrid Roll

HPC Grid

SGE

Java

Ninf-G

SCE

Site Setup for ThaiGridSite Setup for ThaiGrid

• Setup host certificates for Grid• Metadata Service Setup• Enable some services

– Jobmanager-sge– Grid-FTP– GSI-SSH

• Joining with Grid Control Center– Currently using SCMSWeb for central web-base monitoring– Technical contact of each site will send an e-mail with URL of

cluster monitoring nodes– Grid meta-scheduler on GCC will distribute jobs to each

institutes cluster• Setup Grid user

Site SetupSite Setup

• Traditional Grid setup are based on manual setup– Complicate– Error prone

• ROCKS 4.1 provide mechanism for automatic grid installation and configuration– Rocks has already configured almost every services for basic

Grid system

• Anyway, some additional manual setup is needed• We currently use a well-documented admin guide to

provide a step-by-step to join ThaiGrid

Thai Grid Control CenterThai Grid Control Center

• Grid Control Center Software– Monitoring

• Automatic testing• Service Monitoring• Accounting• System monitoring• Netwwork Bandwidth

– Grid-level Job scheduling

Thai Grid Control Center Thai Grid Control Center (con’t)(con’t)• Automatic Grid services status monitoring• Command line for immediate status checking

http://images.google.com/imgres?imgurl=http://www.cnn.com/ALLPOLITICS/1997/09/29/mitchell/mitchell.gif&imgrefurl=http://www.cnn.com/ALLPOLITICS/1997/09/29/mitchell/&h=310&w=300&sz=35&tbnid=satcB_vt8v7nvM:&tbnh=112&tbnw=108&hl=en&start=36&prev=/images%3Fq%3Dit%2527s%2Balive%26start%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26rls%3DDVXA,DVXA:2005-37,DVXA:en%26sa%3DN

Site CertifySite Certify

• Site certification program– Level 1: Network firewall

compliance– Level 2: Middleware

compliance (MPI/G2, Ninf-g)

– Level 3: Job scheduling compliance

– Level 4: Data Grid compliance

• Standard– Define clear software

stack

ThaiGrid Firewall StatusThaiGrid Firewall Status

Overall : 15Pass : 12Partial : 1Fail : 2

ThaiGrid Middleware ThaiGrid Middleware StatusStatus

Overall : 15Pass : 4Partial : 1Fail : 10

Grid ApplicationsGrid Applications

• ThaiGrid Drug Discovery Infrastructure• Grid based data assimilation using RS data• Implicit Narrowband Level Set Method for Fire

Spread Model• The Parallel Simulation Model for Thin Film Dep

osition Using the DSMC Method• Medical Grid

• Access Grid

ThaiGrid Drug Discovery ThaiGrid Drug Discovery InfrastructureInfrastructure

• Partners:– Dr. Jak Saengma, KU– Somsak Sriprayoonsakul, TNGC– IBM Thailand

• Challenges– From over 10000 active compounds available

from Thai medicinal plants database, find a smallest set of compounds that has a potential to be used as a drug

– Very compute intensive. Several month of computing time

• Solution: Use grid to increase computing power to 10-100 times

– Distributed screening process to hundred of hosts on the Grid

• Benefit– Speeding up time to results– Allow scientists to spend more time analyzing

results

Grid based data assimilation Grid based data assimilation using RS datausing RS data

• Dr. Kiyoshi Honda, AIT• Dr. Putchong Uthayopas, THNC• Challenges

– Using GA based data assimilation for agricultural monitoring

– Very long computation time from months to years

– Large amount of RS data needed to be moved around and process

• Solution: – Using Grid to harvest more

computing power– Hide data assimilation process

behind the service using MPI, GridRPC (ninf-g)

• Benefit– Speeding up the calculation

User

Portal

Cluster A

Cluster B Cluster C

Host DSWAP-SERVICE

SWAP-SERVICE

SWAP-SERVICESWAP-SERVICE

Grid

Implicit Narrowband Level Implicit Narrowband Level Set Method for Fire Spread Set Method for Fire Spread Model Model

Dr. Pallop HuabsomboonMahidol University

• Challenges– Simulation of large scale firespred

for forest fire– Very time consuming process

• Solution– Distribute the computation on the

Grid

• Benefit– Expect 10x faster simulation speed– Ability to model much larger

problem within a reasonable time

• Part of TNGP/ Hewlett Packard Join Research Program

The Parallel Simulation Model for Thin FiThe Parallel Simulation Model for Thin Film Deposition Using the DSMC Methodlm Deposition Using the DSMC Method

Dr. Rardchawadee SilapuntDr. Tiranee AchalakulKMUTT, Thailand• Thin film deposition process is

used for many manufacturing industries including very large scale integrated circuits, optics, magnetic, hard and corrosion resistant coatings, micro-mechanics.

• Grid system will be used to accelerate the computation and

simulation process • Part of TNGP/ Hewlett

Packard Join Research Program

simulation domain S

ub

-D

om

ain

Processor

Processor

Processor

MP

IM

PI

Su

b-

Do

ma

inS

ub

-D

om

ain

Medical GridMedical GridSuwachai Sieanoon, Somkiat

Chatchuenyot, Dr. Panjai Tantasnawong

Computer Science Department, Silpakorn University

• To build and develop a distributed information management system using DataGrid technology.

• Apply the distributed information management system to test on exchanging and managing the Patient Records System, which is record complexity and, different format and large size of data.

Grid Services(Internet)Option

Hospital_E

Grid Services

MetadataHospital_A

MetadataHospital_B

MetadataHospital_C

Data Agent

Hospital_F

Hospital_D

GRID Services

Data Agent

Service Registr

y

Meta Data(HL7)

XML

IMAGE

Access Grid Access Grid

• First APEC Virtual Symposium on Pandemic Preparedness using Access Grid, January 20, 2006

• Presentation from Medical Organization in many economies– China, Taiwan, USA,

Vietnam, Thailand, Singapore, Philippine

• Access Grid are used as a key communication infrastructure

Future WorkFuture Work

• Grid Infrastructure– Include “Grid ZeroConf” roll to auto configure the

cluster to join ThaiGrid • Install additional CA certificates• Generate host certificate request• Modify yum repository

• Grid Application– Drug Design application will be deployed fully on

ThaiGrid clusters before July– GIS-base Grid-RPC application due on August– More application coming

End End

Additional SlidesAdditional Slides

Technical ContactTechnical Contact

• Each ThaiGrid member institute need to assign at least one person as technical contact point for all ThaiGrid activity

• Technical contact tasks– Software setup (both cluster and Grid)– Contact point for people inside institute

• Our communication medium are mailing-list, webboard, wiki, skype, and telephone– http://rswg.thaigrid.or.th are the central point for all

technical contacts

Grid: What and WhyGrid: What and Why• Grid is a technology that incorporates distributed

computing systems into a massive computing and storage pool with efficient resource sharing.

• Grid can strengthen national competitiveness in ICT industries as well as promote sciences and engineering research.

• Grid benefits several application domains:– E-commerce, E-government, E-learning– Digital Media and Animation– Geoinformatics, Bioinformatics– Financial Analysis– Etc.

Solving network problemSolving network problem

• General problem– Firewall Compliance– Dealing with smart intrusion detection system– Running Grid on Shared REN/Service network– Network underutilized, unbalance bandwidth

• Develop a network measurement infrastructure – Collaboration PSU/TNGC/KU/Uninet

• Collaboration with network provider (Uninet)

Network Bandwidth OUT (Mbps)

0

5

10

15

20SPIRIT(CU)

GRIDCMU(CMU)

ENQUEUE(KMITNB)

INCA(KMUTT)

SRITHAN(KKU)RDCGRID1(NECTEC)

RDCGRID2(NECTEC)

PSU-GRID(PSU)

SUTGRID(SUT) Network Bandwidth IN (Mbps)

0

20

40

60

80SPIRIT(CU)

GRIDCMU(CMU)

ENQUEUE(KMITNB)

INCA(KMUTT)

SRITHAN(KKU)RDCGRID1(NECTEC)

RDCGRID2(NECTEC)

PSU-GRID(PSU)

SUTGRID(SUT)

ThaiGrid Central ThaiGrid Central MonitoringMonitoring• ThaiGrid currently use OpenSCE to monitor the whole

ThaiGrid• OpenSCE is installed during Rocks installation using

SCE Roll• Each cluster provides monitored information through

HTTP protocol to central monitoring site

Grid Collector

RRD Database Grid

Configure file

Grid Summary Database

Grid Summary

xml file

Network

Presentation Layer

Collector Layer

System API Layer

Web

Summary XML Display

HistoryConfigurati

on

System Informatio

n

Grid Viewer

SCMS

Cluster Collector

Error Notification Email

XML Display

XML Display

Network

scms-getinfo

Summary xml file

SCE config file

Uptime Analysis

Network

Globus

Scheduler

Probe Jobs

driver