1 Peter Sloot: Computational Science, University of Amsterdam, The Netherlands. Interactive Problem Solving: The Polder Meta Computing Inititiative Peter

1Peter Sloot: Computational Science, University of Amsterdam, The Netherlands.

Interactive Problem Solving: The Polder Meta Computing Inititiative

Peter Sloot

Computational Science

University of Amsterdam, The Netherlands


Ariadne’s Red-Rope

– From PSE to Virtual Laboratory and Motivation

– Architecture• Infrastructure• Job Level: Hierarchical Scheduling• Resource management: Task-migration

– Interaction && Case implementation

– Interactive Algorithms


Virtual Laboratory Environment

Internet 2 Wide Area Network

ViSE

Net Client App. User MRI/CT

Distributed Computing & Gigabit Local Area Network

Local User

Local User

Physical apparatus

Virtual-lab Information Management for Cooperation (VIMCO)

Communication & collaboration (ComCol)

Virtual Simulation & Exploration Environment (ViSE)

Advanced Scientific Domains

Computational Physics

System Engineering

Computational Bio-medicine


Interactive Computing: Why?– Goal: From Data, via Information to Knowledge

– Complexity: Huge data-sets, complex processes

– Approach: Parametric exploration and sensitivity analyses:• Combine raw (sensory) data with simulation• Person in the loop:

• Sensory interaction • Intelligent short-cuts


Intro: Case study from biomedicine...


Changing the Paradigm

In Vivo

In Vitro

In Silico



In Vivo

In Vitro

In Silico



In Vivo

In Vitro

In Silico


Current Situation

Observation

Diagnosis & Planning

Treatment


New Possibilities in the VL

Fast, High-throughputLow Latency

Internet

High Performance Super Computing

• Time and Space Independence

• 3D Information

• Simulation based planning

• Surgeon ‘in the loop’


Experimental set-up


Architecture


Architecture Continued: Hybrid system

– Host: The DAS• 24 node parallel cluster in a

200 node wide area machine

• 200 MHz Pentium Pro

• Myrinet 150MB/s

• ATM wide-area interconnect between clusters

9 10 11

8

7

6

141312

15

16

17

2 1 0

3

4

5

212223

20

19

18

GRAPE1 GRAPE0

ATM

Origine 2000

Cave


Immersive Environments


3D Information and Interaction


Problem: Curse of dynamics:

Static task load Dynamic task load

Static resource load Dynamic resource load

Static task allocation Predictable

reallocationDynamical reallocation


Solution To Curse

– Performance of a parallel program usuallydictated by slowest task• Task resource requirements and available resources

both vary dynamically• Therefore, optimal task allocation changes• Gain must exceed cost of migration

– Resources used by long-running programs may be reclaimed by owner


Dynamite Initial State

Two PVM tasks communicating through a network of daemonsMigrate task 2 to node B

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

PVMtask 2


Prepare for Migration

Create new context for task 2Tell PVM daemon B to expect messages for task 2Update routing tables in daemons (first B, then A, later C)

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Newcontext


Checkpointing

Send checkpoint signal to task 2Flush connectionsCheckpoint task to disk

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Newcontext


Cross-cluster checkpointing(design)

Send checkpoint signal to task 2Flush connections, close filesCheckpoint task to disk via helper task

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

Program

PVM

Ckpt

Helpertask


Restart Execution

Restart checkpointed task 2 on node BResume communicationsRe-open & re-position files

Node A Node B

Node C

PVMDA

PVMDB

PVMDC

PVMtask 1

NewPVMtask 2


Special considerations

– Preserve communication• PVM should continue to run as if nothing happened• Use location independent addressing

– Open files• Preserve open file state


Performance

– Migration speed largely dependent on the speed of shared file system• and that depends mostly on the network

– NFS over 100 Mbps Ethernet• 0.4 s < Tmig < 15 s for

2 MB < sizeimg < 64 MB

– Communication speed reduced due to added overhead• 25% for 1 byte direct messages

• 2% for 100 KB indirect messages


Current status: Dynamite Part

– Checkpointer operational under• Solaris 2.5.1 and higher (UltraSparc, 32 bit)

• Linux/i386 2.0 and 2.2 (libc5 and glibc 2.0)

– PVM 3.3.x applications supported and tested• Pam-Crash (ESI) - car crash simulations• CEM3D (ESI) - electro-magnetics code• Grail (UvA) - large, simple FEM code• NAS parallel benchmarks• BloodFlow

– MPI and socket (Univ. of Krakow) libraries available– Scheduling not yet satisfactory


Architecture: Revisited


Design Considerations

– High Quality presentation

– High Frame rate

– Intuitive interaction

– Real-time response

– Interactive Algorithms

– High performance computing and networking...


Problem: Time, time what has become of us?


Solution: Asynchronicity


A police officer to guide the asynchronous processes


Runtime Support

– Need generic framework to support modalities

– Need interoperability

– High Level Architecture (HLA):• data distribution across heterogeneous platforms• flexible attribute and ownership mechanisms• advanced time management


Provoking a bit…

Progress in natural sciences comes from taking things apart ...

Progress in computer science comes from bringing things together...


Proof is in the pudding...

– Diagnostic Findings

• Occluded right iliac artery

• 75% stenosis in left iliac artery

• Occluded left SFA

• Diffuse disease in right SFA


Problem: From Image to Simulation

MR Scan of Abdomen MR Scan of Legs


Solution: 3DManual initialization

Place start point

Wave propagates from start- to end point

Place one or more end points

Backtrack = first estimation of the

centerline

Distance Transform from vessel wall to center centerline

Wave propagates from ‘centerline’ vessel

wall


Wavefront PropagationPlace start point




centerline



wall


MRA: BacktrackPlace start point




centerline



wall


MRA: Wavefront PropagationPlace start point




centerline



wall


MRA: Distance TransformPlace start point




centerline



wall


3-D selection of region of interest


Tracking the vessels


Building the Geometric Models


VR-Interaction


Alternate Treatments

Angio w/ Fem-Fem &

Fem-Pop

AFB w/ E-S Prox.

Anast.

Angio w/Fem-Fem

AFB w/ E-E Prox.

Anast.

Preop


Problem: Flow through complex geometry

– After determining the vascular structure simulate the blood-flow and pressure drop…

– Conventional CFD methods might fail:• Complex geometry• Numerical instability wrt interaction• Inefficient shear-stress calculation


Solution to interactive flow simulation

– Use Cellular Automata as a mesoscopic model system:• Simple local interaction• Support for real physics and heuristics• Computational efficient


Mesoscopic Fluid Model

– Fluid model with Cellular Automata rules

– Collision: particles reshuffle velocities

– Imposed Constraints• Conservation of mass• Conservation of momentum• Isotropy

Details...


...Equivalence with NS– For lattice with enough symmetry: equivalent to the continuous incompressible Navier-Stokes equations:

uuu

u

2 1

0

Pt

u

Implicit parallel and complex geometry support.


Efficient Calculation of Shear-Stress

AND the momentum stress tensor that is linearly related to the shear stresses

i

if i

iif cu

i

iiif ccΠ

x

u

x

u

~

Perpendicular momentum transfer:

From LBE scheme:


Velocity Magnitude

10 cm/sec

0 cm/sec


Peak Systolic Pressures - Rest

150 mmHg

50 mmHg

Angio w/ Fem-Fem &

Fem-Pop

AFB w/ E-S Prox.

Anast.

Angio w/Fem-Fem

AFB w/ E-E Prox.

Anast.

Preop


… last slides...


Internet and Web Software

Distributed Computer infrastructure

Central-part Virtual Laboratory

Physical Apparatus

User

ViSE ComCol VIMCO

Internet and Web Software

Distributed Computer infrastructure

Central-part Virtual Laboratory

User

Computing in Physics

VL for Material Science

ViSE ComCol VIMCO

Meta data Integration

Computing in Engineering

Traffic Payment for mobility

Combining problem solving & data intensive

environments

Bio-medicalComputation

Study of blood flow through

veins

Integration of simulation &

visualization by man in the loop

Bio- informaticsEnvironment

DNA Research

Combing datamining & intelligent data bases

Cultural Inheritance

Environment

Art objects preservationrestoration

Collaborative data

integration

Computing in Engineering

Apply VL in non-quality of service

environment

Modeling VL in non-QoS situation

environment

Other Virtual Laboratory Applications @ UvA


AcknowledgementsStanford:

Charley Taylor, PhD.

Christopher K. Zarins, PhD. M.D.

UvA:

Robert Belleman

Alfons Hoekstra, PhD

Dick van Albada, PhD

Benno Overeinder, PhD

Krakow

Marian Bubak, PhD

Kamil Iskra

RUL/AZL:

H. Reiber, PhD.

Bloem, PhD, M.D.

SARA:

A. de Koning, PhD.

Arcobel:

S. ten Den

IBM:

J. Geise


Support

ICES-KIS-1

ICES-KIS-2

KNAW

NWO/FOM

IBM

SARA

SGI

Platform HPCN


http://science.uva.nl/~sloot

[email protected]



1955 1965 1975 1985 19950.1

1

10

100

1000

10.000

100.000

2005

MFlop/s

1000.000

IBM 704

CDC 6600

Cray X-MP

Cray Y-MP

CM-5

ASCI-RedASCI-Blue

2D Plasma

48 hr Weather

72 hr Weather

Pharmaceutical

?

Structural Biology

Oil reservoir


Results - Mean Flow Rates (ml/min) - Rest

Artery Preop AFB/E-S AFB/E-E Angio/FF Angio/FF-FP

Abd. Aorta 3009 3460 3460 3460 3460

Aortic Bifurc. 670 307 15 1310 1385

R. Int. Iliac 134 156 212 144 132

L. Int. Iliac 464 148 150 335 360

R. Femoral 216 615 663 444 546

L. Femoral 464 550 781 747 618

R. Popliteal 30 283 309 162 303

L. Popliteal 272 351 362 290 277


Cellular Automata

– 1966 Introduced by John von Neumann

– 1985 Stephen Wolfram suggested CA are capable of Universal Computation

– 1990 Lindgren et al., proved UC in 1D CA


1 0 1 0 0 1 1 0 0t=0

1 1 1 0 1 1 1 0 1t=1

0000

0011

0101

0111

1000

1011

1101

1110

Productie Regel 110

Time Evolution of 1D Cellular Automata 110

Back to Mesoscopic Models


The Lattice Gas model

– Fluid model with Cellular Automata rules

– Collision: particles reshuffle velocities

– Imposed Constraints• Conservation of mass• Conservation of momentum• Isotropy

tntntcn iiiii ,,1, xxx


The Hexagonal Lattice


Collision rules examples

Two body collision N1 AND N4 => N2 AND N5 && N3 AND N6

Three body collision N2 AND N4 AND N6 => N1 AND N3 AND N5

Streaming and Colliding


From LGA to LBM

– Average LGA equation to get continuous values instead of boolean values

– Boltzmann molecular chaos assumption to factorize products in collision operator:

=> Iterate:33and iiiiii NNNNNf

),(),(1

),()1,( tftftftf eqiiiii xxxcx


From Micro Dynamics to Macro Dynamics (1)

– Taylor expansion to get continuous differential operators:

itiii

itiiiti

ff

fff

ee

e

221

221



– Chapman Enskog expansion of equilibrium Distribution Function:

– With imposed constraints:

221ii

eqii ffff

1,0 and 0

and

6

1

)(6

1

)(

6

1

6

1

jff

ff

ii

ji

i

ji

ii

eqi

i

eqi

e

eu



– Multi-scale expansion of time and space derivatives:

– Solve collision/flow equation for different order of

0

:balance)order nd-(2

.0

:balance)order st -(1

0021

1

Su rtrt

rt u

1

2 and 21

rttt


Back to mesoscopic models

Documents

1 Peter Sloot: Computational Science, University of Amsterdam, The Netherlands. Interactive Problem Solving: The Polder Meta Computing Inititiative Peter