3D P

aral

lel F

EM (I

II)Pa

ralle

l Vis

ualiz

atio

n us

ing

ppO

pen-

MA

TH/V

IS

Ken

go N

akaj

ima

Tech

nica

l & S

cien

tific

Com

putin

g I (

4820

-102

7)S

emin

ar o

n C

ompu

ter S

cien

ce I

(481

0-12

04)

Par

alle

l FE

M

ppO

pen-

HPC

Ope

n So

urce

Infr

astr

uctu

re fo

r Dev

elop

men

t an

d Ex

ecut

ion

of L

arge

-Sca

le S

cien

tific

A

pplic

atio

ns w

ith A

utom

atic

Tun

ing

(AT)

Ken

go N

akaj

ima

Info

rmat

ion

Tech

nolo

gy C

ente

rM

asak

i Sat

oh (A

OR

I/U. T

okyo

), Ta

kash

i Fur

umur

a (E

RI/U

. Tok

yo)

Hiro

shi O

kuda

(GS

Fro

ntie

r Sci

ence

s/U

. Tok

yo),

Take

shi I

was

hita

(AC

CM

S/K

yoto

U.)

Hid

e S

akag

uchi

(JA

MS

TEC

)

Key

-Issu

es fo

r A

ppl’s

/Alg

orith

ms

tow

ards

Post

-Pet

a &

Exa

Com

putin

gJa

ck D

onga

rra

(OR

NL/

U. T

enne

ssee

) at I

SC

201

3

•H

eter

ogen

eous

/Hyb

rid A

rchi

tect

ure

•C

omm

unic

atio

n/S

ynch

roni

zatio

n R

educ

ing

Alg

orith

ms

•M

ixed

Pre

cisi

on C

ompu

tatio

n•

Aut

o-Tu

ning

/Sel

f-Ada

ptin

g•

Faul

t Res

ilien

t Alg

orith

ms

•R

epro

duci

bilit

y of

Res

ults

3

ppO

pen-

HPC

•pp

Ope

n-H

PC

is a

n op

en s

ourc

e in

frast

ruct

ure

for d

evel

opm

ent

and

exec

utio

n of

opt

imiz

ed a

nd re

liabl

e si

mul

atio

n co

de o

n po

st-p

eta-

scal

e (p

p) p

aral

lel c

ompu

ters

bas

ed o

n m

any-

core

ar

chite

ctur

es, a

nd it

con

sist

s of

var

ious

type

s of

libr

arie

s, w

hich

co

ver g

ener

al p

roce

dure

s fo

r sci

entif

ic c

ompu

tatio

n.

•S

ourc

e co

de d

evel

oped

on

a P

C w

ith a

sin

gle

proc

esso

r is

linke

d w

ith th

ese

libra

ries,

and

the

para

llel c

ode

gene

rate

d is

op

timiz

ed fo

r pos

t-pet

a-sc

ale

syst

ems.

The

targ

et p

ost-p

eta-

scal

e sy

stem

is th

e P

ost T

2K S

yste

m.

•pp

Ope

n-H

PC

sup

ports

app

roxi

mat

ely

2,00

0 us

ers

of th

e su

perc

ompu

ter s

yste

m in

the

Uni

vers

ity o

f Tok

yo, e

nabl

ing

them

to s

witc

h fro

m h

omog

eneo

us m

ultic

ore

clus

ters

to th

e P

ost T

2K S

yste

m.

•pp

Ope

n-H

PC

is d

evel

oped

by

SC

D/IC

D a

nd c

olla

bora

tors

, and

a

five-

year

pro

ject

(FY

.201

1-20

15) s

uppo

rted

by J

apan

ese

gove

rnm

ent (

7 in

stitu

tes,

>30

peo

ple,

5 P

D’s

).

4

Supe

rcom

pute

rs in

U.T

okyo

2 bi

g sy

stem

s, 6

yr.

cycl

e FY 05

0607

0809

1011

1213

1415

1617

1819

Hita

chi S

R11

000/

J218

.8TF

LOP

S, 1

6.4T

B

Fat n

odes with

 large mem

ory

(Flat) M

PI, goo

d comm. perform

ance

京(=K)

Peta

Turning po

int to Hy

brid Parallel Prog. M

odel

Fujit

su P

RIM

EHPC

FX1

0ba

sed

on S

PAR

C64

IXfx

1.13

PFL

OP

S, 1

50TB

Hita

chi S

R16

000/

M1

base

d on

IBM

Pow

er-7

54.9

TFL

OP

S, 1

1.2

TBOur last SMP, to

 be sw

itche

d to M

PP

Hita

chi H

A80

00 (T

2K)

140T

FLO

PS

, 31.

3TB

Post

T2K

>30

PFLO

PS

5

Post

T2K

Sys

tem

>

30 P

FLO

PS

, FY

.201

5-20

16

JCA

HP

C (J

oint

Cen

ter f

or A

dvan

ced

Hig

h P

erfo

rman

ceC

ompu

ting)

: U. T

suku

ba &

U. T

okyo

ht

tp://

jcah

pc.jp

/

Man

y-co

re b

ased

(e.g

. Int

el M

IC/X

eon

Phi

)

MP

I + O

penM

P +

X

ppO

pen-

HP

C h

elps

sm

ooth

tran

sitio

n of

use

rs (>

2,

000)

to n

ew s

yste

m

6

ppO

pen-

HPC

: ppO

pen-

APP

L•

ppO

pen-

HP

C c

onsi

sts

of v

ario

us ty

pes

of optimized

libra

ries,

whi

ch c

over

s va

rious

type

s of

pro

cedu

res

for

scie

ntifi

c co

mpu

tatio

ns.

•pp

Ope

n-A

PP

L/FE

M, F

DM

, FV

M, B

EM

, DE

M•

Line

ar S

olve

rs, M

at. A

ssem

ble,

AM

R.,

Vis

ualiz

atio

n et

c.•

writ

ten

in F

ortra

n 20

03 (C

inte

rface

is a

vaila

ble

soon

)•

Sou

rce

code

dev

elop

ed o

n a

PC

with

a s

ingl

e pr

oces

sor i

s lin

ked

with

thes

e lib

rarie

s, a

nd g

ener

ated

pa

ralle

l cod

e is

opt

imiz

ed fo

r pos

t-pet

a sc

ale

syst

em.

•U

sers

don

’t ha

ve to

wor

ry a

bout

opt

imiz

atio

n tu

ning

, pa

ralle

lizat

ion

etc.

•P

art o

f MP

I, O

penM

P, (

Ope

nAC

C)

7

ppO

pen-

HPC

cov

ers

…88

9

Fram

ewor

kA

ppl.

Dev

.

Mat

hLi

brar

ies

Aut

omat

icTu

ning

(AT)

Syst

emSo

ftwar

e

Sche

dule

of P

ublic

Rel

ease

(w

ith E

nglis

h D

ocum

ents

, MIT

Lic

ense

)ht

tp://

ppop

enhp

c.cc

.u-to

kyo.

ac.jp

/

•R

elea

sed

at S

C-X

Y (o

r can

be

dow

nloa

ded)

•M

ultic

ore/

man

ycor

ecl

uste

r ver

sion

(Fla

t MP

I, O

penM

P/M

PI

Hyb

rid) w

ith d

ocum

ents

in E

nglis

h•

We

are

now

focu

sing

on

MIC

/Xeo

n P

hi•

Col

labo

ratio

ns w

ith s

cien

tists

are

wel

com

e

His

tory

•S

C12

, Nov

201

2 (V

er.0

.1.0

)•

SC

13, N

ov 2

013

(Ver

.0.2

.0)

•S

C14

, Nov

201

4 (V

er.0

.3.0

)

10

New

Fea

ture

s in

Ver

.0.3

.0ht

tp://

ppop

enhp

c.cc

.u-to

kyo.

ac.jp

/•

ppO

pen-

AP

PL/

AM

R-F

DM

: AM

R

fram

ewor

k w

ith a

dyn

amic

load

-ba

lanc

ing

met

hod

for v

ario

us F

DM

ap

plic

atio

ns•

HA

CA

pKlib

rary

for H

-mat

rix c

omp.

in

ppO

pen-

AP

PL/

BE

M–

Aki

hiro

Ida

(Kyo

to U

.)

11

•U

tiliti

es fo

r pre

-pr

oces

sing

in

ppO

pen-

AP

PL/

DE

M

12

Col

labo

ratio

ns, O

utre

achi

ng•

Col

labo

ratio

ns–

Inte

rnat

iona

l Col

labo

ratio

ns•

Law

renc

e B

erke

ley

Nat

iona

l Lab

.•

Nat

iona

l Tai

wan

Uni

vers

ity•

IPC

C（In

tel P

aral

lel C

ompu

ting

Cen

ter）

•O

utre

achi

ng, A

pplic

atio

ns–

Larg

e-S

cale

Sim

ulat

ions

•G

eolo

gic

CO

2S

tora

ge•

Ast

roph

ysic

s•

Ear

thqu

ake

Sim

ulat

ions

etc

.•

ppO

pen-

AT,

ppO

pen-

MA

TH/V

IS,

ppO

pen-

MA

TH/M

P, L

inea

r Sol

vers

–In

tl. W

orks

hops

(201

2, 2

013)

–Tu

toria

ls, C

lass

es

13

ppO

pen-

MA

TH

•A

set

of c

omm

on n

umer

ical

libr

arie

s–

Mul

tigrid

sol

vers

(ppO

pen-

MA

TH/M

G)

–P

aral

lel g

raph

libr

arie

s (p

pOpe

n-M

ATH

/GR

AP

H)

–P

aral

lel v

isua

lizat

ion

(ppO

pen-

MA

TH/V

IS)

–Li

brar

y fo

r cou

pled

mul

ti-ph

ysic

s si

mul

atio

ns (l

oose

-co

uplin

g) (p

pOpe

n-M

ATH

/MP

)•

Orig

inal

ly d

evel

oped

as

a co

uple

r for

NIC

AM

(atm

osph

ere,

un

stru

ctur

ed),

and

CO

CO

(oce

an, s

truct

ured

) in

glob

al c

limat

e si

mul

atio

ns u

sing

K c

ompu

ter

–B

oth

code

s ar

e m

ajor

cod

es o

n th

e K

com

pute

r. »

Pro

f. M

asak

i Sat

oh (A

OR

I/U.T

okyo

): N

ICA

M»

Pro

f. H

iroya

su H

asum

i (A

OR

I/U.T

okyo

): C

OC

O

•D

evel

oped

cou

pler

is e

xten

ded

to m

ore

gene

ral u

se.

–C

oupl

ed s

eism

ic s

imul

atio

ns

14

NICAM:

Semi-Unstructured Grid

CO

CO

: Tri-

Pol

ar F

DM

ppOpen-MATH/MP

Coupler

•Gri

d Tr

ansf

orm

atio

n•

Mul

ti-E

nsem

ble

•IO

•Pre

-and

pos

t-pr

oces

s•F

ault

tole

ranc

e•M

×N

Post-Peta-Scale

System

-System S/W

-Architecture

Ocean Model

Atmospheric

Model-1

NIC

AM-

Agrid

NIC

AM-

ZMgr

id

Regional Ocean Model

Non Hydrostatic Model

J-cu

p

MIR

OC-

A

COCO

Regi

onal

CO

COM

atsu

mur

a-m

odel

MIROC-A:

FDM/Structured Grid

Atmospheric

Model-2

15

c/o

T.A

raka

wa,

M

.Sat

oh

FDM: Seism

ic W

aveProp

agation 

Mod

el size:80x80

x400

 km

Time：

240 s

Resolutio

n (space)：

0.1 km

 (regular)

Resolutio

n (tim

e)  ：

5 ms

(effe

ctive freq

.<１Hz

）

FEM: B

uilding Re

spon

se 

Mod

el size：

400x40

0x20

0 m

Time：

60 s

Resolutio

n (space)：

1 m

Resolutio

n (tim

e)   : 1

 ms

ppOpe

n‐MAT

H/M

P: Spa

ce‐tem

poral interpo

latio

n, M

apping

 be

tween FD

M and

 FEM

 mesh, etc.

Lar

ge-Scal

e C

oupl

ed

Sim

ula

tions

in F

Y.2

013

Challenge (FY201

3) : A test of a

 cou

pling sim

ulation of FDM

 (regular 

grid) and

 FEM

 (uncon

structed

 grid

) usin

g ne

wly develop

ed ppO

pen‐

MATH/MP Co

upler

16

c/o

T.Fu

rum

ura

17

ppO

pen-

MA

TH/V

IS•

Par

alle

l Vis

ualiz

atio

n us

ing

Info

rmat

ion

of

Bac

kgro

und

Vox

els

[Nak

ajim

a &

Che

n 20

06]

–FD

M v

ersi

on is

rele

ased

: ppO

pen-

MA

TH/V

IS-F

DM

3D•

UC

Dsi

ngle

file

•P

latfo

rm–

T2K，

Cra

y–

FX10

–Fl

at M

PI

•U

nstru

ctur

ed/H

ybrid

ver

sion

–N

ext r

elea

se[R

efin

e]AvailableMemory

= 2.0

Avai

labl

e mem

ory

size (

GB)

, not

avai

labl

e in

this

vers

ion.

MaxVoxelCount

= 500

Max

imum

num

ber o

f vox

els

MaxRefineLevel

= 20

Max

imum

num

ber o

f ref

inem

ent l

evel

s

18

Sim

plifi

ed P

aral

lel V

isua

lizat

ion

usin

g B

ackg

roun

d Vo

xels

•

Oct

ree-

base

d A

MR

•A

MR

app

lied

to th

e re

gion

whe

re g

radi

ent o

f fie

ld

valu

es a

re la

rge

–st

ress

con

cent

ratio

n, s

hock

wav

e, s

epar

atio

n et

c.•

If th

e nu

mbe

r of v

oxel

s ar

e co

ntro

led,

a s

ingl

e fil

e w

ith 1

05m

eshe

s is

pos

sibl

e, e

ven

thou

gh e

ntire

pr

oble

m s

ize

is 1

09w

ith d

istri

bute

d da

ta s

ets.

SD

SC

201

3

19

FEM

Mes

h (S

W J

apan

Mod

el)

SD

SC

201

3

20

Voxe

l Mes

h (in

itial

)

SD

SC

201

3

21

Voxe

l Mes

h (2

-leve

l ada

pted

)

SD

SC

201

3

22

Exam

ple

of S

urfa

ce S

impl

ifica

tion

Initi

al

(11,

884

tri’s

)50

% re

duct

ion

(5,9

42 )

95%

redu

ctio

n (5

94)

98%

redu

ctio

n(2

38)

SD

SC

201

3

23

pFEM

3D +

ppO

pen-

MA

TH/V

ISpF

EM

-VIS

Files

>$ cd <$O-fem2>

>$ cp/home/z30088/fem2/C/fem3dv.tar .

>$ tar xvffem3dv.tar

>$ cd <$O-fem2>/fem3dv/sr

>$ make

>$ cd ../run

>$ pjsubgo.sh

New Files

>$ cd <$O-fem2>

>$ cd fem3d/parallel/pmesh

>$ cp/home/z30088/pmesh.f.

>$ cd ../src

>$ cp/home/z30088/input_grid_p.c.

>$ cd ../part

>$ cp/home/z30088/main.f.

24

Mak

efile

pFE

M-V

IS

PPOHVISDIR= /home/z30088/fem2/ppohVIS_PFEM3DSTR_2014_12_22/ppohVIS/

LIB_DIR = $(PPOHVISDIR)/lib

INC_DIR = $(PPOHVISDIR)/include

CFLAGSL = -I/home/z30088/ppohVIS_test/include

LDFLAGSL = -L/home/z30088/ppohVIS_test/lib

LIBSL = -lppohvisfdm3d

OPTFLAGS = -Kfast

-I$(INC_DIR)

LFLAGS = -L$(LIB_DIR) $(LIBS) -lm

# .SUFFIXES:

.SUFFIXES: .o .c

.c.o:

$(CC) $(OPTFLAGS) -c $< -o $@

# TARGET = ../run/sol

default: $(TARGET)

OBJS =¥test1_p.o pfem_init.o...

$(TARGET): $(OBJS)

$(CC) $(OPTFLAGS) -o $@ $(OBJS) $(LFLAGS)

clean:

/bin/rm

-f *.o *~ $(TARGET)

25

<$O-fem2>/fem3dv/run

pFE

M-V

IS

../pmesh/pcube.0

../pmesh/pcube.1

../pmesh/pcube.2

../pmesh/pcube.3

../pmesh/pcube.4

../pmesh/pcube.5

../pmesh/pcube.6

../pmesh/pcube.7

go.sh

INPUT.DAT

vis.cnt

../pmesh/pcube

1 0

2 2000

#!/bin/sh

#PJM -L "node=1"

#PJM -L "elapse=00:10:00"

#PJM -L "rscgrp=lecture"

#PJM -g "gt85"

#PJM -j

#PJM -o "cube8.lst"

#PJM --mpi "proc=8"

mpiexec ./sol

26

pFE

M3D

+ p

pOpe

n-M

ATH

/VIS

pFE

M-V

IS

<HEADER>.*

Dis

trib

uted

Loc

al

Mes

h Fi

les

sol

with ppohVIS

INPUT.DAT

vis_disp.1.inp

vis_sigma-zz.1.inp

Para

View

Out

put

vis.cnt

27pF

EM

-VIS

#include <stdio.h>

#include <stdlib.h>

FILE* fp_log;

#define GLOBAL_VALUE_DEFINE

#include "pfem_util.h"

#include "ppohVIS_PFEM3D_Util.h"

extern void PFEM_INIT(int,char**);

extern void INPUT_CNTL();

extern void INPUT_GRID();

extern void MAT_CON0();

extern void MAT_CON1();

extern void MAT_ASS_MAIN();

extern void MAT_ASS_BC();

extern void SOLVE33();

extern void RECOVER_STRESS();

extern void PFEM_FINALIZE();

intmain(intargc,char* argv[])

{double START_TIME,END_TIME;

structppohVIS_BASE_stControl*pControl=NULL;

structppohVIS_BASE_stResultCollection

*pNodeResult=NULL;

PFEM_INIT(argc,argv);

ppohVIS_PFEM3D_Init(MPI_COMM_WORLD);

INPUT_CNTL();

pControl= ppohVIS_PFEM3D_GetControl("vis.cnt");

INPUT_GRID();

if(ppohVIS_PFEM3D_SetMeshEx(

NP,N,NODE_ID,XYZ, ICELTOT,ICELTOT_INT,ELEM_ID,ICELNOD,

NEIBPETOT,NEIBPE,IMPORT_INDEX,IMPORT_ITEM,EXPORT_INDEX,EXPORT_ITEM)) {

ppohVIS_BASE_PrintError(stderr);

MPI_Abort(MPI_COMM_WORLD,errno);

};

C/m

ain

(1/2

)

28pF

EM

-VIS

SOLVE33();

RECOVER_STRESS();

pNodeResult=ppohVIS_BASE_AllocateResultCollection();

if(pNodeResult

== NULL) {

ppohVIS_BASE_PrintError(stderr);

MPI_Abort(MPI_COMM_WORLD,errno);

};

if(ppohVIS_BASE_InitResultCollection(pNodeResult, 4)) {

ppohVIS_BASE_PrintError(stderr);

MPI_Abort(MPI_COMM_WORLD,errno);

};

pNodeResult->Results[0] =

ppohVIS_PFEM3D_ConvResultNodeItem3N(NP,"disp",X);

pNodeResult->Results[1] =

ppohVIS_PFEM3D_ConvResultNodeItemPart3N(NP,0,"sigma-xx",SIGMA_N);

pNodeResult->Results[2] =

ppohVIS_PFEM3D_ConvResultNodeItemPart3N(NP,1,"sigma-yy",SIGMA_N);

pNodeResult->Results[3] =

ppohVIS_PFEM3D_ConvResultNodeItemPart3N(NP,2,"sigma-zz",SIGMA_N);

if(ppohVIS_PFEM3D_Visualize(pNodeResult,NULL,pControl,"vis",1)) {

ppohVIS_BASE_PrintError(stderr);

MPI_Abort(MPI_COMM_WORLD,errno);

};

ppohVIS_PFEM3D_Finalize();

PFEM_FINALIZE() ;

C/m

ain

(2/2

)

29

vis.

cnt

pFE

M-V

IS

[Refine]

Section for Refinement Control

AvailableMemory = 2.0

（GB）not in use

MaxVoxelCount = 1000

Max Voxel #

MaxRefineLevel = 20

Max Voxel Refinement Level

[Simple]

Section for Simplification Control

ReductionRate = 0.0

Reduction Rate of Surf. Patches

[Outpt]

Output Format

FileFormat = 1

=1:ParaView, =2:MicroAVS(OLD)

=3:MicroAVS(NEW)

32,7

68 e

lem

ents

, 39,

949

node

s 55

4 el

emen

ts, 8

28 n

odes

D

efor

mat

ion

Fig.

not

sup

porte

d.