Scientific Visualization at JSC · 30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome Folie 1 Jens Henrik Göbbert1 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany

Folie 1EoCoE-F3F Meeting - Rome30.11.-02.12. 2016

Jens Henrik Göbbert1

1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany

Cross-Sectional-Team „Visualization“

[email protected]

30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome

Scientific Visualization at JSC

Mitglie

d d

er

Helm

holtz-G

em

ein

schaft


Visualization at JSCCross-Sectional Team “Visualization”

Domain-specific User Support and Research at JSC

Scientific Visualization– R&D + support

for visualization of scientific data

Virtual Reality– VR systems

for the analysis and presentation

Multimedia– multimedia productions

for websites, presentations

or on TV

http://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Visualization/_node.html


12x Visualization Nodes

2 GPUs Nvidia Tesla K40 per node

12 GB RAM on each GPU

2x Vis. Login Nodes

jurecavis.fz-juelich.de

10x Vis. Batch Nodes

8 nodes with 512 GB RAM

2 nodes with 1024 GB RAM

Keep in mind:

Visualization is NOT limited to

Visualization nodes (incl. GPUs) only.

(software rendering is possible on any node)

Visualization at JSCGeneral Hardware Setup

http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JURECA/UserInfo/VisNodes.html?nn=1538466

Data

GPFS

vis login

nodes

vis batch

nodes

compute

nodes

JURECA

10x

2x

InfiniB

and

1872x

12xlogin

nodes


Visualization at JSC

– Cross-Sectional Team “Visualization”

– Setup - Software & Hardware

Simplify your life: Remote 3d Visualization

In Situ Visualization

– Introduction

– Coupling Strategies

– Example

Outline


Mitglie

d d

er

Helm

holtz-G

em

ein

schaft

Simplify your life:

Remote 3D Visualization

on JURECA with VNC


Why remote visualization …

… makes sense for the user.

Do not spend time installing & configuring visualization software.

Do not move your data.

Do not be satisfied with your local hardware.

… makes sense for CS-Team „Visualization“.

Install & configure visualization software once for all users.

Any new feature is available to all users.

Fix bugs once for all users.

Motivationlowering the barriers to visualization

More time left for the interesting things.


nice blue JSC background clock

counting

up/down

MOTD windowCPU, memory

utilization

GPU utilization

VNC utilization

desktop

symbels for

vis apps,

LLview, …

visualization application

Motivationlowering the barriers to visualization


User’s Work-

station

vis login node:

- direct user access

- no accounting

- shared with other users

- no parallel jobs (no srun)

vis batch node:

- access via batch system

- accounting

- exclusive usage

- parallel jobs possible

Data

GPFS

vis login

nodes

vis batch

nodes

compute

nodes

JURECA

10x

2x

InfiniB

and

1872x

12xlogin

nodes

Remote 3D VisualizationGeneral Setup

Firewall


Remote 3D Visualizationwith X forwarding + Indirect Rendering

Traditional Approach (X forwarding + Indirect Rendering)

ssh –X <USERID>@<SERVER>

uses GLX extension to X Window System

X display runs on user workstation

OpenGL command are encaplusated inside X11 protocol stream

OpenGL commands are executed on user workstation

disadvantages User´s workstation requires a running X server.

User´s workstation requires a graphic card capable of the requied OpenGL.

User´s workstation defines the quality and speed of the visualization.

User´s workstation requires all data needed to visualize the 3d scene.

Try to AVOID for 3D visualization.


State-of-the-Art Approach (VNC with VirtualGL)

vncserver, vncviewer

platform independent

application independent

session sharing possible

advantages No X is required on user´s workstation (X display on server, one per session).

No OpenGL is required on user´s workstation (only images are send).

Quality of visualization does not depend on user´s workstation.

Data size send is independent from data of 3d scene.

http://www.virtualgl.org

Remote 3D Visualizationwith VNC (Virtual Network Computing) + GLX forking

Try to USE for 3D visualization.


GLX forking

vglrun (VirtualGL)

OpenGL applications send both GLX and

X11 commands to the same X display.

Once VirtualGL is preloaded into an

OpenGL application, it intercepts the

GLX function calls from the application

and rewrites them.

The corresponding GLX commands are

then sent to the X display of the 3d X

server, which has a 3D hardware

accelerator attached.

Disadvantages:

⁻ Multiple users on a login node share the same graphic cards.

⁻ No resource management

for memory/compute of graphic cards.

⁻ CPU encodes the VNC stream.

⁻ Currently no hardware accelerated image encoding.

Remote 3D Visualizationwith VNC (Virtual Network Computing) + GLX forking

http://www.virtualgl.org


… manually startup VNC connection1. ssh to HPC system

2. authenticate via SSH key pair

3. look for an existing VNC server

(you may want to reconnect)

4. submit job via slurm

5. wait for job to start

start VNC server on node

6. establish SSH tunnel

7. setting up SSH agent forwarding

8. start TurboVNC client and connect

Remote 3D VisualizationSimplify the access

Data

GPFS

vis login

nodes

vis batch

nodes

compute

nodes

JURECA

10x

2x

InfiniB

and

1872x

12xlogin

nodes

Firewall

https://trac.version.fz-juelich.de/vis/wiki/vnc3d/manual

This is far too time consuming.


… automatically startup VNC

ScienTific Remote Desktop Launcher (Strudel)

– Windows, OS X, Linux

– written in Python

– developed by

• Monash University, Melbourne, Australia

Remote 3D VisualizationSimplify the access

Data

GPFS

vis login

nodes

vis batch

nodes

compute

nodes

JURECA

10x

2x

InfiniB

and

1872x

12xlogin

nodes

https://trac.version.fz-juelich.de/vis/wiki/vnc3d/strudel

One-Click solution.

Firewall





[email protected]


Mitglie

d d

er

Helm

holtz-G

em

ein

schaft

Invitation:

Try JURECA Visualization Nodes


InvitationTry JURECA Visualization Nodes - 30.11.16 – 09.12.16

VNC with VirtualGL (using Strudel)


download & install

• SSH (Putty for Windows)

• TurboVNC

• Strudel

configure

• download SSH keys

• https://fz-juelich.sciebo.de/index.php/s/qec7SKVOEBDenpH• get passphrase for one of the SSH keys

• [email protected]

• load SSH key to your SSH agent

run & play

• start Strudel and connectPlease follow the instructions from



https://fz-juelich.sciebo.de/index.php/s/qec7SKVOEBDenpH

mailto:[email protected]



InvitationTry JURECA Visualization Nodes - 30.11.16 – 09.12.16

VNC with VirtualGL (using Strudel)


Keep in mind:

• SSH key agent must be running on your system.

• SSH key for JURECA must be loaded into the SSH key agent.

Please send us your Feedback.

[email protected]



Try JURECA Visualization NodesOSPRay

http://www.ospray.org/

http://www.sdvis.org/ospray/download/talks/IEEEVis2016_OSPRay_talk.pdf

CPU ray tracing framework

for scientific vis. rendering

efficient rendering on CPUs

ray tracing / high fidelity rendering

made for scientific visualization

Built on top of

Embree (Intel ray tracing kernels)

Intel SIMD Program Compiler

Integrated into

ParaView, VMD, VisIt, VL3,

EasternGraphics,...

ParaView

OSPRay

“FIU” Ground Water Simulation

Texas Advanced Computing Center (TACC) and Florida International University


Try JURECA Visualization NodesOSPRay with ParaView

http://www.ospray.org/

http://www.sdvis.org/ospray/download/talks/IEEEVis2016_OSPRay_talk.pdf

Ray tracing within ParaView

Build option in ParaView 5.2 by default

Why ray tracing?

gives more realistic results

adds “depth” to your image

can be faster on large data

Requirement:

CPUs: Anything SSE4 and newer

(in part, including Intel©

Xeon Phi™ Knights Landing)

Cooperation with Electrochemical Process Engineering (IEK-3)

Jülich Forschungszentrum GmbH, Germany

ParaView

(standard renderer)

ParaView

(OSPRay)


Try JURECA Visualization NodesParallel ParaView





[email protected]


Mitglie

d d

er

Helm

holtz-G

em

ein

schaft



Why do we need In Situ Visualization ?Visualization “the old way”

JUQUEEN, @ FZ Jülich Data Archive, @ FZ

Jülich

Jülich Storage Cluster

(JUST), @ FZ Jülich

@ AIA, RWTH Aachen University

Why changing a running System …. ?

JURECA, @ FZ Jülich

wikipedia.org, Chevron


Why do we need In Situ Visualization ?Bottlenecks for Visualization

Example:

AIA, RWTH Aachen University

Large Eddy Simulation

Analysis of Chevron Geometries

123 TB (= 30 days to download with 50 MB/s)

@ AIA, RWTH Aachen University

X

Moving data between computing centers

becomes a major bottleneck.


Why do we need In Situ Visualization ?Bottlenecks for Visualization

XX

Moving data to the storage devices

will become a major bottleneck.


What is In Situ Visualization ?Basics

Data compression

Reduce I/O traffic and volume.

Access to more data.

Post-processing

In situ processingDump Times:

Enable real-time simulation

exploration.

Reduce latency to first result.

Avoid large-scale post-processing.

…..

…..

Visualizing results of a running simulations …

… without the need to copy the data to a storage device.


What is In Situ Visualization ?Basics

But ….

Simulation- and Visualization code …

… might have different internal data structures.

… might scale best with different parallelization strategies.

Simulation- and Visualization developer …

… might have different priorities.

… might not work in the same team.

Visualizing results of a running simulations …

… without the need to copy the data to a storage device.

Coupling simulation- and visualization codes is a challenge.


What is In Situ Visualization ?Staged vs. On-Node

„Staged“ In Situ Visualization

(or in transit visualization)

„On-Node“ In Situ Visualization


What is In Situ Visualization ?Staged vs. On-Node

„Staged“ In-Situ Visualization „On-Node“ In-Situ Visualization

Advantages:

No need to copy data between nodes

or clusters.

Zero-copy possible.

Simpler to implement.

Disadvantages:

Hardware resources (cpu / memory)

need to be shared between simulation

and visualization.

Advantages:

No need to share hardware resources

(cpu / memory) between simulation and

visualization.

Disadvantages:

Data needs to be copied between

nodes or even clusters.

No zero-copy possible.

More difficult to implement.


What is In Situ Visualization ?Loose vs. Tight

„Loose“ Coupling „Tight“ Coupling

Advantages:

Simpler to implement.

Single executable.

Zero-copy possible.

Disadvantages:

Bug in visualization code might affect

simulation.

„On-Node“ in situ visualization by

design.

Advantages:

Less dependencies at compile/link time

Two separate applications.

Disadvantages:

Data transfer between Simulation and

Visualization more difficult to

implement.

Zero-copy difficult.


Major barriers to in situ visualization are …

… first,

the individual implementation-, optimization- and coupling-costs

to integrate the needed functionality to each simulation code and setup

can often not be justified.

… second,

the usage of in situ visualization requires much training for scientists

who's research work in general does not focus on visualization in the first

place.

Lowering the barriers to in situ visualizationWhat are the major barriers?


Direct Approchcoupling simulation code to in-situ visualization

sim. developer

simulation

VisIt/

Libsim

ParaView/

Catalyst

vis. developer

ParaView/Catalyst

VisIt/Libsim


Damaris - http://damaris.gforge.inria.fr

SENSEI - http://www.sensei-insitu.org

Strawman -

GLEAN -

JUSITU -

Adapter Approachcoupling simulation code to in situ visualization

sim. developer vis. developer

simulation

VisIt/

Libsim

ParaView/

Catalyst

http://damaris.gforge.inria.fr/

http://www.sensei-insitu.org/



simulation

VisIt/

Libsim

ParaView/

Catalyst

data adapter bridge analyze adapter



data adapter bridge analyze adapter





[email protected]


Mitglie

d d

er

Helm

holtz-G

em

ein

schaft


Example


… psOpen

JUSITU

flow solver (LES, DNS)

multiphysics, multiscale

finite volume method

lattice-Boltzmann method

discontinuous Galerkin method

level-set for surface tracking

Lagrangian particle solver

Institute of Aerodynamics Aachen

RWTH Aachen University, Germany

C++11 + MPI + OpenMP + GPU

… CIAO … ZFS

flow solver (LES, DNS)

multiphysics, multiscale

structured finite difference

method

level-set for surface tracking

level-set/volume-of-fluid interface

Lagrange particle solver

tabulated/finite rate chemistry

overset mesh refinement

moving meshes

Institute for Combustion

Technology

RWTH Aachen University,

Germany

Fortran90 + MPI

JUQUEEN BigWeek participant

flow solver (DNS)

highly resolved turbulence

pseudo-spectral approach

Institute for Combustion Technology

RWTH Aachen University, Germany

Chair of Num. Thermo-Fluid Dyn.

TU Freiberg, Germany

Fortran90 + MPI + OpenMP

JUQUEEN BigWeek participant

Applications lately coupled with VisIt/Libsim


Why In Situ Visualization – An example.spray injection & droplet formation


Baumgarten, C., Mixture formation in internal combustion engines, Springer, 2006.

Aye, M.M. et al., Studying the influence of k-factor of different nozzles on spray and

combustion under diesel engine-like conditions in a high pressure spray chamber (submitted).

Geometry of injection systems

increasingly complex

Emission and pollutant formation

strongly depend on injection systems

Experimental investigation of geometry

impact very challenging smaller

k-factorlarger

k-factor

Ra

tio o

f so

ot

tra

ns.

inte

nsity

High-fidelity spray simulations required!



Experimental investigation of

injection systems very challenging

due to small length (~100 μm) and

high velocity scales (~600 m/s).

Highly resolved simulations result in

large amount of data which can be

reduced with in-situ visualization.

Predictive simulations using in-situ

visualization and studying injection

systems and resulting cavitation

give more insight.

Le Chenadec, V. and Pitsch, H., “A 3D Unsplit Forward/Backward Volume-of-Fluid Approach and Coupling to the Level Set Method,” Journal of Computational Physics 233:10-33, 2013

Le Chenadec, V. and Pitsch, H., “A monotonicity preserving conservative sharp interface flow solver for high density ratio two-phase flows,” Journal of Computational Physics 249:185-203, 2013




Dump data with high temporal and spatial resolution needed.


Application CIAO on JUQUEEN (specific setup)

■ excellent scaling of solver

■ file output is bottleneck at scaledump data with high temporal resolution not possible

=> In situ visualization helps!



Summary & Conclusion

Try the visualization nodes at JSC

Send me your feedback.


Size of simulation data is rising constantly

and I/O becomes a major bottleneck.

A huge number of simulation codes and scientists will

benefit from in situ visualization (in situ processing).

But, decisions have to be made

how to coupling simulation and analysis tools in future.


Questions … ?

rendered with Blender from a DNS of a diesel injection spray of ITV, RWTH Aachen University

Documents

Scientific Visualization at JSC · 30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome Folie 1 Jens Henrik Göbbert1 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany