Visualization of Particle In Cell Simulations19671/FULLTEXT01.pdfVisualization of Particle In Cell Simulations Författare Author Patric Ljung Sammanfattning Abstract A numerical simulation

ExamensarbeteLITH-ITN-MT-EX--00/001--SE

Visualization ofParticle In Cell Simulations

Patric Ljung

2000-03-28

Department of Science and technology Institutionen för teknik och naturvetenskapLinköping University Linköpings universitetSE-601 74 Norrköping, Sweden 601 74 Norrköping

LITH-ITN-MT-EX--00/001--SE

Visualization ofParticle In Cell Simulations

Examensarbete utfört i Medieteknikvid Linköpings Tekniska Högskola, Campus Norrköping

Patric Ljung

Handledare: Anders YnnermanExaminator: Anders Ynnerman

Norrköping 2000-03-28

Avdelning, InstitutionDivision, Department

Institutionen för teknik och naturvetenskap

Department of Science and technology

DatumDate

2000-03-28

SpråkLanguage

Svenska/SwedishEngelska/English

RapporttypReport category

Examensarbete

B-uppsatsC-uppsatsD-uppsats

URL för elektronisk version

http://www.ep.liu.se/exjobb/itn/2000/mt/001

ISBN

ISRN LITH-ITN-MT-EX--00/001--SE

Serietitel och serienummerTitle of series, numbering

TitelTitle

Visualization of Particle In Cell Simulations

FörfattareAuthor

Patric Ljung

SammanfattningAbstract

A numerical simulation case involving space plasma and the evolution of instabilities thatgenerates very fast electrons, i.e. approximately at half of the speed of light, is used as a testbed for scientific visualisation techniques. A visualisation system was developed to provideinteractive real-time animation and visualisation of the simulation results. The work focuseson two themes and the integration of them. The first theme is the storage and managementof the large data sets produced. The second theme deals with how the Visualisation Systemand Visual Objects are tailored to efficiently visualise the data at hand.The integration of the themes has resulted in an interactive real-time animation and visual-isation system which constitutes a very powerful tool for analysis and understanding of theplasma physics processes. The visualisations contained in this work have spawned many newpossible research projects and provided insight into previously not fully understood plasmaphysics phenomena.

NyckelordKeywords

Scientific Visualization, Volume rendering, 3D Textures, Computational Plasma Physics,High Throughput Data Storage

Visualisation of Particle In Cell SimulationsDiploma work report

Patric LjungScientific Visualisation Group

Department of Science and TechnologyLinköpings universitet

March 28, 2000

ii

Abstract

A numerical simulation case involving space plasma and the evolution of instabili-ties that generates very fast electrons, i.e approximately at half of the speed of light,is used as a test bed for scientific visualisation techniques. A visualisation systemwas developed to provide interactive real-time animation and visualisation of thesimulation results. The work focuses on two themes and the integration of them.The first theme is the storage and management of the large data sets produced. Thesecond theme deals with how the Visualisation System and Visual Objects are tai-lored to efficiently visualise the data at hand. The integration of the themes hasresulted in an interactive real-time animation and visualisation system which con-stitutes a very powerful tool for analysis and understanding of the plasma physicsprocesses. The visualisations contained in this work have spawned many new pos-sible research projects and provided insight into previously not fully understoodplasma physics phenomena.

iii

iv

Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Report outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Definitions and prerequisites . . . . . . . . . . . . . . . . . . . .4

2 Problem Description 72.1 The Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 The Simulation Case: Cosmic Rays . . . . . . . . . . . . . . . .82.3 Particle In Cell Simulations . . . . . . . . . . . . . . . . . . . . .11

2.3.1 The Simulation Cycle . . . . . . . . . . . . . . . . . . .122.3.2 Simulation Complexity . . . . . . . . . . . . . . . . . . .122.3.3 Particles and Cells . . . . . . . . . . . . . . . . . . . . .13

3 Data Management 173.1 Key Features . . . . . . . . . . . . . . . . . . . . . . . . . . . .173.2 Data Organisation . . . . . . . . . . . . . . . . . . . . . . . . . .183.3 Data Representation . . . . . . . . . . . . . . . . . . . . . . . . .203.4 Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . .213.5 Direct I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.6 The FORTRAN 77 Data Set Format . . . . . . . . . . . . . . . .233.7 Disks, File Systems and Data Set Sizes . . . . . . . . . . . . . . .24

4 Visualisation of Data 254.1 Key Features . . . . . . . . . . . . . . . . . . . . . . . . . . . .254.2 Visualisation System Design . . . . . . . . . . . . . . . . . . . .264.3 Data Mapping and Transformation . . . . . . . . . . . . . . . . .274.4 Visual Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . .284.5 1D - Ribbon Graph . . . . . . . . . . . . . . . . . . . . . . . . .314.6 2D - Height Field Rendering . . . . . . . . . . . . . . . . . . . .33

4.6.1 Normal Calculation for Height Field . . . . . . . . . . . .344.7 3D - Volume Rendering . . . . . . . . . . . . . . . . . . . . . . .374.8 3D - Particle Rendering . . . . . . . . . . . . . . . . . . . . . . .40

v

vi CONTENTS

5 Results and Performance 435.1 Interfacing Simulation and Visualisation . . . . . . . . . . . . . .435.2 Data Set Management and Streaming . . . . . . . . . . . . . . . .445.3 The Visual Objects . . . . . . . . . . . . . . . . . . . . . . . . .445.4 Physical Results . . . . . . . . . . . . . . . . . . . . . . . . . . .47

5.4.1 Unmagnetised Plasma . . . . . . . . . . . . . . . . . . .475.4.2 The Surfatron . . . . . . . . . . . . . . . . . . . . . . . .48

6 Conclusions 516.1 Critics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52

A Onyx2 RealityMonster 55A.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55A.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

B Source Files 61B.1 Source Code Organisation . . . . . . . . . . . . . . . . . . . . .61

B.1.1 Patric’s Common Library . . . . . . . . . . . . . . . . . .61B.1.2 Patric’s Data Visualiser . . . . . . . . . . . . . . . . . . .61B.1.3 Patric’s Volume Visualiser . . . . . . . . . . . . . . . . .62

C FORTRAN 77 Binary Data Set File Format 65C.1 Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65C.2 Slice Records . . . . . . . . . . . . . . . . . . . . . . . . . . . .67

D Colour Images 69

List of Tables

3.1 Element Types . . . . . . . . . . . . . . . . . . . . . . . . . . . .213.2 Cell Composition . . . . . . . . . . . . . . . . . . . . . . . . . .213.3 Data Set Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

5.1 Sliced Volume Rendering Performance . . . . . . . . . . . . . . .455.2 Fill-rate versus Volume download . . . . . . . . . . . . . . . . .465.3 3D Point Rendering Performance . . . . . . . . . . . . . . . . . .46

A.1 Onyx2 InfiniteReality Performance - Single pipe with 4 RasterManagers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56

C.1 Index File Header . . . . . . . . . . . . . . . . . . . . . . . . . .66C.2 Cell Composition Structure . . . . . . . . . . . . . . . . . . . . .66C.3 Index Record Structure . . . . . . . . . . . . . . . . . . . . . . .67

vii

viii LIST OF TABLES

List of Figures

1.1 The Kanitza triangles . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Visualisation Principle . . . . . . . . . . . . . . . . . . . . . . . 21.3 Domain and Range forf . . . . . . . . . . . . . . . . . . . . . . 4

2.1 Supernova Remnant: ASCA Image of SN 1006 . . . . . . . . . .92.2 X-ray Spectrum of SN 1006 . . . . . . . . . . . . . . . . . . . .102.3 Particle In Cell Single Simulation Cycle . . . . . . . . . . . . . .132.4 Particle In Cell Simulation Box . . . . . . . . . . . . . . . . . . .142.5 Particle In Cell Cyclic Border Conditions . . . . . . . . . . . . .142.6 Bilinear Interpolation . . . . . . . . . . . . . . . . . . . . . . . .15

3.1 Data Set File Organisation . . . . . . . . . . . . . . . . . . . . .18

4.1 Visualisation Main Loop and Callbacks . . . . . . . . . . . . . .264.2 Structured and Unstructured Data . . . . . . . . . . . . . . . . .284.3 Load, Generate, and Render Pipeline . . . . . . . . . . . . . . . .314.4 Interleaved Graphics Data . . . . . . . . . . . . . . . . . . . . .324.5 Ribbon Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . .324.6 Height Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.7 Height Field Principle . . . . . . . . . . . . . . . . . . . . . . . .344.8 Second Degree Curve . . . . . . . . . . . . . . . . . . . . . . . .364.9 Density Volume . . . . . . . . . . . . . . . . . . . . . . . . . . .374.10 Volume Slicing for Orthographic and Perspective Projection . . .394.11 Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41

D.1 Super Nova Remnant SN1006 . . . . . . . . . . . . . . . . . . .70D.2 Electron Velocities . . . . . . . . . . . . . . . . . . . . . . . . .71D.3 Second Beam and Empty Island . . . . . . . . . . . . . . . . . .72D.4 Ribbon Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . .73D.5 Height Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74D.6 Density Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . 75D.7 Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76D.8 Particles and Height-Field Serie 1 . . . . . . . . . . . . . . . . .77D.9 Particles and Height-Field Serie 2 . . . . . . . . . . . . . . . . .78

ix

x LIST OF FIGURES

D.10 Particles and Height-Field Serie 3 . . . . . . . . . . . . . . . . .79D.11 Volume Images Serie 1 . . . . . . . . . . . . . . . . . . . . . . .80D.12 Volume Images Serie 2 . . . . . . . . . . . . . . . . . . . . . . .81

Chapter 1

Introduction

1.1 Background

Most models describing real world physical processes can not be dealt with usinganalytical tools. Instead one must rely on Numerical Simulations. A numericalsimulation is based on the fundamental laws and relations which govern the be-haviour of the system and can be defined mathematically.

In an explicit simulation, i.e. reflecting an evolution in time. A physical caseis set up and the simulated objects are given an initial state1. A computer programis then used to move the system forward in time using the mathematical equationsdeduced from the laws and relations. The forward stepping in time is then repeatedself-consistently until certain criteria are fulfilled. In order to reach a result aboutthe state at some point in time, one must step through time until the point of inter-est is reached. If some parameters or conditions are changed, the whole simulationmust be run from the beginning. The run-time for simulations may vary from a fewseconds to several weeks, limiting the number of possible runs. Hence, one needsto pick specific cases, gain insight and then re-couple to analytical understanding.But it is essential to save as much information as possible from the simulationsto be examined and analysed after the simulation. This becomes particularly im-portant when one is not certain what to find. Thus, it is necessary to deal withincreasingly large data sets. The results from numerical simulations are in mostcases not individual numbers, but come in terms of data structures. Within thelarge data sets produced, traces or evidence of important facts can be embedded.In simulations or “real” experiments physical properties of a process often show upas structures; topological features within the data sets. This leads us to use a “tool”with an excellent capability to detect such features: the human vision system. Itscapabilities to detect data properties are vast, both for spatial and temporal proper-ties. Representing the data by means of graphics and visual objects is probably thefastest way to communicate large amounts of data from a computer system to the

1State refers to specific values of the dynamic properties for an object, e.g. position, speed, anddirection for a car

1

2 CHAPTER 1. INTRODUCTION

human brain, in many cases relying on the human brain’s capability of detectingstructures from incomplete information. In figure 1.1 an example of how the brainis capable to detect shapes with incomplete information is shown [8].

Figure 1.1: The Kanitza triangles

Visualisation is a means of obtaining insight into a problem through the useof visual objects. These visual objects can be generated from various classes ofdata. Data can be mapped in many ways ton-dimensional visual objects withdifferent properties like colour, surface type, transparency, motion, etc. The datacan be abstract, e.g. non-Cartesian. For example, one dimension can be time, theother velocity, and the third spatial position, animated along increasing temper-ature. Topological properties are important for the understanding of processes.Scientific Visualisation thus deals with research in which the feedback providedby the visualisation process is important. Scientific Visualisation has been used inNatural Science, Medicine, and Engineering, but is also useful in other areas, suchas Human Behaviour, Economics, and Philosophy. Figure 1.2 shows a principle ofvisualisation.

Experiment

Archive

Computation

Process ControlPost Processing

ImageComputation Control

Insight!ComputerRepresentation

ImageGeneration

Figure 1.2: Visualisation Principle

Plasma simulation is one example which challenges Scientific Visualisation,both by means of handling large data sets and of producing visual objects. Aplasma consists of ionised gas, and is also sometimes referred to as the fourthstate of matter. The other three states are the Solid, Liquid, and Gas states. Inthe universe, most matter is in the plasma state, e.g. stars consists of plasma. In

1.2. REPORT OUTLINE 3

a plasma the charged particles interact with electric and magnetic fields. In theplasmas studied as cases in this diploma work, there can be about1010 particles inconsideration2, on the scale of some ten cubed meters. Analytic approximationsexist to model plasma physics in the linear domain. But for the many non-linearcases, a numerical simulation is currently the only possible way.

1.2 Report outline

This report is divided into six chapters and four appendices:

Chapter 1: Introduction The introduction contains a brief background and moti-vation for the work. A descriptive outline of this report and some definitionsare also included.

Chapter 2: Problem Description The Problem Description chapter introduces thegoal for this diploma work. We here provide the theory to the physical sim-ulation case and the simulation process.

Chapter 3: Data Management for Interactive Streaming This chapter discussesproblems and solutions related to handling and storage of the large data setsproduced by the numerical simulations.

Chapter 4: Visualisation of Data from PIC Simulations In this chapter the de-sign of the visualisation system is introduced and how it is connected withmanagement of large data sets. The visual objects designed to visualise thesimulation data are explained.

Chapter 5: Physical Results and Performance MeasurementsThis chapter con-tains the results of this work. Both in terms of software, computer graphics,and performance as well as physical results and the use of visualisation inphysics and numerical simulations.

Chapter 6: Conclusions This final chapter contain conclusions and suggestionsfor future work as well as some short comings.

Appendix A: Onyx2 RealityMonster This appendix describes the hardware andsystem software used in the project. Also included are performance mea-surements.

Appendix B: Source Files This appendix contains a list of classes and filenameswith explanations, so one can find where classes are defined.

Appendix C: FORTRAN 77 Binary Data Set File Format This appendix givesa detailed description of the binary storage format for the data sets used inthis work.

2The number of molecules in one litre of water is100018

6.022 · 1023 ≈ 3.4 · 1025. Each moleculeconsist of 18 protons, 16 neutrons and 18 electrons, i.e. 36 charged particles.


Appendix D: Colour Images contains colour images of some of the figures andseries of screen shots.

1.3 Definitions and prerequisites

It is assumed that the reader of this document has a fundamental understanding ofthree-dimensional computer graphics. This document does not explain homoge-neous coordinates, transformation matrices, common texture mapping, and othersimilar issues. Knowledge about OpenGL (Open Graphics Library) is also useful[19]. Please refer to the following books for more information: Computer Graphics- Principles and Practice [6], Virtual Reality Systems [16]. Neither are fundamentalcomputer issues like operating systems, programming languages, etc., explained.Suggested reading is Operating System Concepts [15], Introduction to AutomataTheory, Languages, and Computation [9], and Distributed Systems [3].

To the benefit of the reader some of the key concepts and terms used throughoutthis thesis document are defined below.

Node A Node is in this document an equivalent to a CPU in the used hardware. Incomputation context, one process is often locked to one CPU, which is thenfully dedicated to the execution of that process.

Ordo The complexity of an algorithm or communication model is described usingdifferent measures. The Ordo notionO defines the upper limit of the com-plexity. If an algorithm is of the complexity2n2+700n+20 it belongs to thecomplexity classO(n2). Whenn increases the dominant part is2n2. Con-stants are ignored in theoretical discussions since they are highly dependenton implementations.

Dimension and dimensionality refers to the size of vectors in the DomainDf

and RangeRf for a functionf : Df → Rf , see figure 1.3. Outer and Innerdimension are other used terms describing the dimension of elements inDf

andRf , respectively. Multidimensional specifies two or more elements inthe domain elementsx ∈ Df . Multivariate specifies two or more elementsin the range elementsy ∈ Rf . Applying a functionf to a setDf to get a setRf is sometimes called mappingDf to Rf .

Df Rf

x y

y = f(x)

Figure 1.3: Domain and Range forf

1.3. DEFINITIONS AND PREREQUISITES 5

Data Set is a set of ordered elements in some domain. Unstructured data refersto a more or less random order where the elements themselves contain theirDomain position, like spatial location. Structured data impose an outer order,for example the position in the set structure determines the spatial location,and the elements in the set belong only to the RangeRf .

Data Slice refers to a slice of elements, a subset of the Data Set. One dimensionis chosen to be sliced up for convenient handling. Often the time dimensionis sliced. This implies the constraint of having at least one dimension in theset ordered. Without any structured dimension, slicing can not be properlydefined. Slices can be of any number of dimensions.

Value is an explicit number of a property for some object. It can further be clas-sified in Scalar Value and Vector Value. A scalar is a single value, e.g.15or 235.44. A vector is ann-tuple ofn scalar values, e.g.(3, 4, 5), (π, e), or(43, 22, 34, 44, 0, 1, 0). The different values of a vector is also referred to aselements. Element order has meaning in a vector, in contrast to a set whereorder has no meaning. The term elements is used both to denote elementswithin a mathematical vector and elements within a set. The elements withina set can be vectors, each with a number of elements.


Chapter 2

Problem Description

2.1 The Task

When I searched for a suitable diploma work for my masters degree, I aimed ata project which would involve state-of-the-art computer graphics. Finally I got intouch with the newly formed Scientific Visualisation Group at the Department forScience and Technology at Linköpings universitet, which could provide me withboth a suitable case and high-end resources for such a project. We then worked outa plan matching my field of interest, it was defined by the following items:

• investigation of control mechanisms, data distribution and data representa-tion in results from Particle In Cell (PIC) codes

• use of 3-dimensional graphics and possibly virtual reality techniques to dis-play subsets of the multidimensional spaces spanned by the data

• development of visualisation tools that enable the user to effectively manip-ulate and interact with the output from the PIC simulations

• use volume rendering techniques for particle functionals

With an emphasis on computer graphics technologies these items involve manyparts from the visualisation principle described in the introduction.

1. How to implement the arrow from Computation to Computer Representa-tion. That is, saving and storing data from the simulation.

2. Design a suitable format and structure for the Computer Representation ofthe simulation data. Design a system that implements the arrow from Com-puter Representation to Image Generation.

3. Define the visual objects to use in the Image Generation.

4. Finally, let the insight gained from the early simulations and versions ofthe software decide directions in simulation and further development of thevisualisation software, the arrows back from Insight.

7

8 CHAPTER 2. PROBLEM DESCRIPTION

The available resources for simulation and graphics production were an SGI Onyx2RealityMonster with 12 R10000 CPUs with an internal memory capacity of 6 GBand 90 GB of striped SCSI-disks for data storage. The RealityMonster consists ofthree InfiniteReality graphics pipes put together in one system. Detailed informa-tion about the system and its performance is given in appendix A.

The rest of this chapter aims at the explanation of the physical background andthe simulation case used in this work. Finally, a section explains the PIC code andthe numerical simulation principle it is based on.

2.2 The Simulation Case: Cosmic Rays

In our project, we have been looking into the origin of cosmic rays. Sources forcosmic rays have been associated with Supernova Remnants (SNRs) [7]. Figure2.1 (D.1 in colour) shows the X-ray radiation from supernova remnant SN 1006.When the star exploded it created a shell of hot plasma expanding into space. Theplasma shell collides with the background plasma in space and a shock is producedin which X-ray light is observed. In figure 2.2 the energy spectrum of the Rim(shock) and the Interior is shown. In the interior there are spectral peaks that canbe associated with certain atomic resonances. In addition, the intensity is lowcompared to that of the rim.

The shock shows quite a different spectrum, which has been shown to resem-ble synchrotron radiation of relativistic electrons orbiting around a magnetic field[7]. This, together with the much higher intensity suggests that the cosmic raygeneration mechanism is linked to the shock. The challenging question is how theradiation from the rim can be generated when the SNR plasma collides with theinterstellar plasma. Since the latter of them does not contain electrons with rela-tivistic velocities. For example, the probability of finding electrons with0.5c in theSolar Wind [2] is practically zero. Since the particles can obtain weakly relativis-tic velocities by crossing through the shock, an acceleration mechanism must bepresent at the shock. A theory that has been suggested involves two stages, eachgiving a contribution of energy to the electrons.

Relativistic electrons can be generated by Fermi acceleration [7]. A phe-nomenological explanation of Fermi acceleration can be given in terms of the fol-lowing analogy. Consider two tennis players playing tennis and assume that:

1. The ball must at least cross the distance∆x between the players.

2. The players’ rackets must be firmly tied, otherwise the energy in the swingwill not be transfered to the ball (inelastic reflection).

3. Energy loss by friction must be smaller than the energy gain from the rackets.

Each time a player hits the ball, its energy is increased. It travels to the opponentfaster than before. The longer they play, the faster the ball goes back and forth,until condition 2 is not fulfilled any more. Transformed to plasma physics, this

2.2. THE SIMULATION CASE: COSMIC RAYS 9

Figure 2.1: Supernova Remnant: ASCA Image of SN 1006


Figure 2.2: X-ray Spectrum of SN 1006

metaphor means that the electrons must pass the plasma shock, the distance∆x.They must also experience some kind of rigid plane to bounce against, which arelow frequency MHD waves (Magneto Hydro Dynamical [2]) [7].

First∆x from item 1 must be found for the shock,∆x is given by the Larmor(or gyro) radius for protons [2]. This is because the shock is determined by protonscales [7]. We define the Larmor radius for electrons and protons to be∆xe and∆xp, respectively. The radii are derived from the Larmor frequencyωc

ωc =e

m| ~B| (2.1)

wherem is the mass of the particle ande is its electric charge. The magnetic fieldis given by ~B. Using the electron massme and proton massmp, we define theLarmor radii.

∆xe =vthe

ωce

(2.1)=me

e| ~B|vthe (2.2)

∆xp =vthp

ωcp

(2.1)=mp

e| ~B|vthp (2.3)

For the Fermi acceleration to take place, we need to have at least∆xe ≥ ∆xp.Using equations 2.2 and 2.3 we get

me

e| ~B|vthe ≥

mp

e| ~B|vthp ⇒ vthe ≥

mp

mevthp (2.4)

2.3. PARTICLE IN CELL SIMULATIONS 11

If both the protons and the electrons are in a thermal equilibrium and we have aMaxwellian distribution [2], we can define the thermal velocitiesvthe andvthp forelectrons and protons as

vthe =

√kTe

mevthp =

√√√√kTp

mp(2.5)

whereTe is the electron temperature in Kelvin andTp is the corresponding protontemperature. The constantk is the Boltzmann constant.

v2the

v2thp

=k

meTe

kmp

Tp

=mpTe

meTp

For plasma in equilibrium we haveTe = Tp so that

v2the

v2thp

=mp

me⇒ v2

the =mp

mev2thp

Squaring the thermal velocity relation in 2.4 gives

v2the ≥

m2p

m2e

v2thp

We setv2thp = v2

thp, thus we find

v2the ≥

m2p

m2e

v2thp =

m2p

m2e

me

mpv2the ⇒ v2

the ≥mp

mev2the

orTe ≥

mp

meTe

This means that the temperatureTe for the electrons must be at leastmp

me≈ 2000

times hotter than that of equilibrium. Or the thermal velocity must be at least√2000 ≈ 45 times faster than that of the equilibrium to give∆xe = ∆xp.

The search for mechanisms which can push the electrons to such high velocitiesis the purpose of the simulations performed in conjunction with this visualisationproject.

2.3 Particle In Cell Simulations

A Particle In Cell (PIC) simulation [4] is a numerical method to resolve states forparticles, magnetic and electric fields over time. The particle state refers to specificvalues of the properties for particles, which in this case is defined in the so calledphase-space. Phase-space is a six-dimensional domain in which elements have aposition vector~x = (x, y, z) and a velocity vector~v = (vx, vy, vz) combined into


(~x,~v). A particle is thus a point in this space. Magnetic and electric field staterefers to the approximated vector field in the simulation grid. The combined statefor all particles and fields are grouped together as the simulation state at a certaintime-step.

2.3.1 The Simulation Cycle

The simulation is given initial physical and simulation parameters and initial par-ticle states are generated by different distributions in the phase-space domain. Thesimulation then iterates by taking the current state, and solve Maxwell’s equations2.6 – 2.9 and the relativistic Lorentz’ equation 2.10.~B is the magnetic flux density,~E electric field,ρ the charge density,~j the current density. The permeability andpermittivity constant are given byµ0 andε0, respectively [2].

∇ · ~E = ρ/ε0 (2.6)

∇ · ~B = 0 (2.7)

∇× ~E = −d ~B

dt(2.8)

∇× ~B = µ0~j + ε0µ0

d ~E

dt(2.9)

d

dt

m~v√1− v2

c2

= e( ~E + ~v × ~B) (2.10)

Discretised versions of these equations are used to calculate fields and currentsfrom the particle states. These fields are then used to calculate the forces on theparticles which are then moved accordingly. The procedure is then iterated in aself-consistent manner. The simulation cycle is shown in figure 2.3.

2.3.2 Simulation Complexity

The current PIC simulation program uses an approximation which differs from areal situation; no individual interactions between particles are considered. Takingparticle collisions into account increases the computational complexity fromO(n)to O(n2), wheren is the number of particles. If particle collisions were to beincluded all particles must be checked for interaction with each other at each time-step. Since one of the strengths of the PIC-code is that it can be parallelised,collision handling imposes not only an increase in computational complexity butalso an increased communication complexity.

The real number of particles involved in the simulated processes is too largefor current super computers. Nonetheless, it is possible to approximate groups ofreal particles with so called Super Particles [4]. Each super particle then representa number of real particles and are orders of magnitudes larger. And from here on,references to particles, i.e. electrons and protons mean their super particle counter-part.


Reduce currents

Statesnot saved

Save states to disk

Compute currents fromparticle states

Compute electric andmagnetic fileds

Compute newvelocities and move

particles

Figure 2.3: Particle In Cell Single Simulation Cycle

2.3.3 Particles and Cells

The physical process takes place in some spatial range. The plasma phenomenastudied in this case are simulated in one spatial dimension. It is important to pointout that the spatial domain is defined as cyclic. This means that if a particle movesbeyond the end of a spatial dimension’s range it is injected back at the beginning ofthe range, i.e. particles are periodically wrapped around and it applies to all spatialdimensions. This excludes structures and phenomena on a larger scale. However,one must be careful when interpreting the results.

The PIC-code can be run in parallel to shorten the total run time for a simula-tion. Each node ofk nodes then works with a fraction (1

k ) of the total number ofparticles. The particles managed by a node is kept at the same node through-outthe entire simulation. A node’s particles are also distributed over the whole spatialdomain for the simulation.

The concept of cells comes from the spatial discretisation of currents and elec-tromagnetic fields, as shown in figure 2.4. The actual simulation box is white,while the grey cells around are the guard cells, implementing the cyclic bordercondition, as shown in figure 2.5. A guard cell picks up a particle and inserts it atthe corresponding mirror cell.

The size of a grid cell is typically set to the Debye length [2]. Within a cell, theparticles contribute a fractional current to each of the vertices. The particle’s chargeand velocity determines the particle’s current contribution to the cell. From theparticle’s position, the current is distributed to the vertices by bilinear interpolation,


Real simulation box cells

Guard cells

Figure 2.4: Particle In Cell Simulation Box

Figure 2.5: Particle In Cell Cyclic Border Conditions


c00n c10n

c01n c11n

u

v pn

x0 x1

y0

y1

Figure 2.6: Bilinear Interpolation

see figure 2.6 and equation 2.11

c00n = c(pn)(1− u)(1− v)

c01n = c(pn)(1− u)v

c10n = c(pn)u(1− v)

c11n = c(pn)uv

u = x(pn)− x0x1 − x0

v = y(pn)− y0y1 − y0

(2.11)

wheren denotes the particle number andx(), y() is the particle’s position inx-direction andy-direction, respectively, andc() denotes a particle’s generated cur-rent. When all particles have distributed their individual current to their cor-responding cell vertices, the summed current for each vertex is reduced from allnodes to a total current at each grid vertex in the real simulation box. All nodesthen compute the electromagnetic field at each vertex from the total current.

In the configuration for the simulation program, we have defined a specificframe of reference. The spatial domain along the x-axis is moving with the centreof mass and current. The phenomena of interest is best analysed with this frame ofreference. Hence, particles that are moving with the same speed as the plasma willappear to stand still. Particles that are faster thus moves in increasing x-directionand slower ones moves in the opposite direction.


Chapter 3

Data Management for InteractiveStreaming

As indicated in the previous chapter, numerical simulations can create vast amountsof data. Experiments and real world measurements are also producing increasinglylarge data sets. Nevertheless,large is a relative concept and must be described inrelation to something. Here the term Large Data Sets is used to refer to data sets toolarge to fit in primary memory. Such a notion is useful since there is a significantdifference in the way we handle sets that fit in memory versus sets that do not fit.Some optimisations can be done if the set is just slightly larger, but in those cases, asmall amount of extra memory could be added and the problem is removed. InsteadI have focused on the cases for which more memory is not a solution.

3.1 Key Features

Facing the challenge of choosing a suitable design for the Data Management Sys-tem (DMS) and Storage Format, there are a number of features that must be orshould be supported. Most of these are comprised in the general concept of Real-Time Visualisation and Animation. More specifically, these features are:

1. Support for reading and delivering data at rates of 10 – 30 slices/s. Eachslice representing one rendered image.

2. Support to increase animation speed by skipping slices and playing forwardand backward, i.e. random access of slices.

3. Handling of data spread across different file systems, in the case when largefile system volumes are not available.

This has resulted in a design in which an index file is used to keep record of all dataslices, and one or more files contain the data slices. Figure 3.1 shows the layout ofthe data set file organisation I have designed.

17

18 CHAPTER 3. DATA MANAGEMENT

HeaderIndex0

Index1

Index2

Index3

Index4

Indexn− 2

Indexn− 1

Slice0

Slice1

Slice2

Slice3

Slice4

Slice5

Index File Slice Pointers Data Files

Figure 3.1: Data Set File Organisation

If files are limited to specific sizes, multiple files can be used. One other im-portant property is that the format is self-contained in the files. This means thatthe format does not require external information on how to load the data, neitherare dimensions of the data externally defined. Using a separate index file, suchinformation is held in one small place, while the large data set can be distributedover a larger space. If the information were stored with the data, reading throughthe actual data set to find information about it would be required.

3.2 Data Organisation

An aspect that has influenced the design is the possible future use of other dataaccess methods than files. It should be possible to use decentralised data and accessthe data by network communication proxies. This would enable computationalsteering or on-line access to simulations on super computers. In this project Ihave implemented support for local files. Nonetheless, the data set class hidesthe current use of files. The slices are accessed by retrieving them individuallyfrom a pdv_DataSet object. The data slices are represented by objects of thepdv_DataSlice class. One of the outer dimensions of the data set is used toslice the data set. This Slicing Dimension is used to address the different slicesin the set. The slices are currently accessed only by slice index, and not with theslicing dimension’s value. The slices are indexed from0 to n − 1. The accessinto the data set is thus made in two steps. First the right slice must be retrieved,and second, the correct element in the current slice is accessed. This access schemeintroduces some constraints for the access to the data set but provides a suitable cueto the DMS about what data that is soon to be used. Another possible scheme whichI have not explored, is to first select a working range for one or more dimensions.This will give a cue to the DMS to load the selected subset. If one wants to have

3.2. DATA ORGANISATION 19

a different dimension for the slicing dimension, this is currently not possible. Apossible work-around is to preprocess the data and change the slicing dimension. AData Organisation scheme that allows run-time reordering of the data set withoutreducing the performance more than noticeable is of great interest, and shouldbe given priority in future work. One advantage of the current scheme is thatindividual slices need not be of the same size. Thus the resolution, number ofelements in a slice can be different for individual slices.

The elements in the slices can be ordered in various ways. In this contextorder means the access order for subsets of the data and how these subsets arereduced until a specific element is left. This defines the storage order for the outerdimensions in the data set. The first dimension in the Domain is used to select aspecific slice. Next is the order within a slice. Three different types are common:

Inner Order means that the elements are ordered with the outmost index takingthe longest steps and the innermost takes the smallest steps. As in the fol-lowing code:

for (i = 0; i < imax; i++) {for (j = 0; j < jmax; j++) {

for (k = 0; k < kmax; k++) {element = data[(i*jmax + j)*kmax + k];

}}

}

Outer Order is the opposite toInner Order and means that the elements areordered with the innermost index taking the longest steps and the outermosttakes the smallest steps. As in the following code:

for (i = 0; i < imax; i++) {for (j = 0; j < jmax; j++) {

for (k = 0; k < kmax; k++) {element = data[(k*jmax + j)*imax + i];

}}

}

Bricking is done by grouping a subset of neighbouring elements in all dimensions.These bricks can then be bricked in macro-bricks such that an hierarchyof bricks is constructed. If the lowest level is a brick ofn3 elements in an × n × n fashion, the second level are macro-bricks ofm3, m × m × mbricks. For example, with two levels of bricks, a90 × 128 × 128 volumeconsists of6 × 8 × 8 macro-bricks, each macro-brick consists of4 × 4 × 4bricks, each brick consisting of4×4×4 elements. Some padding is requiredin the first dimension. Bricks are made to match cache-line size and macro-bricks to match page size in the hardware used. This order has not been used


since its primary benefit comes with ray casting or ray tracing methods forvolume rendering in real-time [13].

The access to the elements are thought of as coordinate to address mappings, wherea coordinate in the Domain is mapped to a memory address where the actual ele-ment is located. These mappings are provided by subclasses topdv_CoordinateToAddressIfc .We have implemented support for both Inner and Outer Order, but not for Brick-ing. There are generic classes for Inner and Outer Order which operates on ar-bitrary number of dimensions. The special cases for two and three dimensions isimplemented by specifically dedicated classes to provide fast mapping.

3.3 Data Representation

In this context Data Representation deals with the storage format of the elementsin a data set. Elements in structured grids and volumes are often referred to asCells. In the source code, a data set element is called a Cell. The Cell could bea simple scalar value or a vector value. Data Representation thus deals with thecomposition of a Cell, but it does not consider the meaning of any data, if thedata under consideration represents spatial position, temperature, etc. In this senseit could be considered as abstract - without any specific interpretation or meaning.Despite the fact that the meaning of the data is not considered, the usage of the datais important since it has impact on issues like performance and storage capacity. Inthe case of real-time visualisation and animation, speed and data transfer are twokey features that must be supported by the system. At the same time, general issuesin terms of good system design and flexibility must be taken into account.

Many simulation programs are written in FORTRAN and often by non com-puter scientists, whose primary interest lies in the simulation and not in program-ming and data storage and management. As a consequence, output data is oftensaved as text data. Which was the case with the PIC code used in this project. Fortext based formats, the size of the data is much larger than if the data is stored in abinary form. And second, the data must be parsed and translated into binary formwhen reading it, and converted from binary to text representation when it is writtento file. Parsing text data takes a substantial time of processing, compared to none(at best) for binary storage.

Initially, the PIC code used in this project produced text output. This was thefirst issue focused on. The output routines in the PIC code was changed to use abinary storage format. The fraction of CPU time used for saving data is low, so thisdo not affect the simulation run-time noticeable. If it would, it is only to the better.A binary format is to some extent hardware dependent, but an absolute necessityin order to achieve fast data handling. The hardware differences can in most casesbe overcome by paying attention to those differences and include the current typein the description of the data. Data can then be converted off-line or on-the-fly.

In the data representation model designed in this work, a data set is a set of

3.4. COMPRESSION 21

Table 3.1: Element Types

Type Size DescriptionDS_BYTE 1 Signed 8 bit integerDS_UBYTE 1 Unsigned 8 bit integerDS_INT16 2 Signed 16 bit integerDS_UINT16 2 Unsigned 16 bit integerDS_INT32 4 Signed 32 bit integerDS_UINT32 4 Unsigned 32 bit integerDS_INT64 8 Signed 64 bit integerDS_UINT64 8 Unsigned 64 bit integerDS_FLOAT32 4 32 bit floating pointDS_FLOAT64 8 64 bit floating pointDS_FLOAT128 16 128 bit floating point

Table 3.2: Cell Composition

Field DescriptionElementOffset Specifies the offset in bytes within a CellElementType Specifies the element type, one of the types specified in

table 3.1.

cells, where each cell contains one or more elements1. For structured data, eachcell corresponds to a mapping from a Domain to the Range, where the cell is amember of the Range. The elements within the cell can be of any type definedby the data management system, see table 3.1. The elements do not need to be ofthe same type within a cell. But all cells must be of the same composition. Thecomposition of the cell is defined globally for a data set. The composition arrayhas the same number of elements as the cell. The structure of the compositionarray elementCellComposition is shown in table 3.2. Access to individualcell elements is provided by the classpdv_CellElementConverter whichtakes the cell composition data and applies it on an address given by subclasses topdv_CoordinateToAddressIfc . The user can extract any cell element bysupplying the current index for the element. And it can be converted to a specificnative type within the system, likeunsigned char , long int , or double .

3.4 Compression

Various compression methods can also be applied. For scientific reasons lossycompression is not preferred in general. But compression also introduces the needfor compression and decompression, which must be handled by the processing re-

1The entities within a data set is also called elements. But here elements refer to the componentsof the cell, similar to vector elements


sources if no hardware compression is available. Considering the read case, com-pression is preferred only when equation 3.1 is satisfied.

TRead > TReadCompressed + TDecompress (3.1)

Whenever reading raw data is slower than reading compressed data and decom-press it, compression is a suitable option. The compressed data size and decom-pression time should also be predictable in real-time systems. Compression wasnot implemented nor investigated in detail in this work.

3.5 Direct I/O

Reading data from files is an important task for the Operating System (OS). It isresponsible for managing the disk usage and to optimise performance for the typeof applications used. This have lead to the use of file buffers [15]. They implementa cache level between the disks and the main memory. Frequently used blocks arethen kept in main memory for fast access. This scheme requires the data to first beread into the OS’s file buffers (system memory) and then copied to user memoryspace.

However, streaming of large data sets is not well suited for the normal filebuffering techniques. Two issues are important:

1. Streaming through a large data set will completely fill the available filebuffers. In a local time frame, the data is never reused so the caching isnot needed, which is hard for the OS to find out.

2. The use of file buffers requires data to be copied from system memory touser memory. With increasing band-width to and from disks, this means thatmore memory must be copied and thus consume more CPU resources.

With the goal to read data with highest possible speed, a feature called DirectI/O is used in our system. Direct I/O bypasses the Operating System’s file buffersby reading data directly to user memory. But, this imposes some constraints on theDMS. Reading and writing of data from disk is always done in complete memorypages, usually of 4 kilobytes. Furthermore, data read and write to files must bealigned on memory page borders. It is an error to read or write data at an offsetdifferent from multiples of the memory page size. Using Direct I/O then allowus to avoid the problems attached with streaming large data sets via normal filehandling.

An alternative to Direct I/O is the use of Memory Mapped Files (MMF). MMFactually provides a simpler interface to the files. By mapping a file to a specificaddress range of the user’s memory, a program can directly read and write into thatmemory area. The OS will then load and save the memory pages of the file asthey are requested. This technique solves the problem of copying data. But sincestreaming of large data sets will fill up internal memory, the problem from item

3.6. THE FORTRAN 77 DATA SET FORMAT 23

one is not solved. The system is responsible for managing what pages that are tobe reused. In other words, the OS is responsible for the caching of the data whilethe program is responsible when Direct I/O is used.

The reason why MMF was not used is the lack of control over memory man-agement. A user controlled scheme can keep down memory usage by reusing thememory for the data slices. On system level, it is hard to determine the use ofthe memory. On the application level, the chance to know what portions of thedata that is required now and in the next step is higher than that on the OS level.And most important, the application most likely know when data is not needed anylonger in the local time span.

3.6 The FORTRAN 77 Data Set Format

A task that was not as easy to implement as was initially thought, was the handlingof the binary format that a FORTRAN program produces. Searching through themanuals and user’s guides for FORTRAN programming, no definition was foundon the binary I/O format used in FORTRAN. From this I have drawn the conclusionthat binary output from FORTRAN programs are seldom used in conjunction withusing the data by a non-FORTRAN program, e.g. a C/C++ program, as in thecase for this work. This opinion has been supported by the people in the groupI have worked with and people at the National Supercomputer Centre (NSC). InFORTRAN, binary files are mostly used to save data and read it back, within thesame program.

Binary files in FORTRAN are constituted of a number of ordered records. Eachrecord may be of arbitrary size and the main operations supported are writing arecord, reading a record, step forward or backward one record, and open and close.One write statement creates one record. All data in a record must be written in asingle write-statement. The real format of the file is not specified, but examinationsof the produced files shows that each record is embraced by a pair of length marks.The record size is prefixed and postfixed to the record. Thus, a double linkedlist is actually implemented. In the structures defined for the binary format, theselength marks have been taken into account and is a part of the format. Althoughthey are not specifically written out in a FORTRAN program, they are handled byother programs. On the computer system used in this work, the FORTRAN binaryfile format produces 32 bit integers as length marks. The format of the begin andend record marks is probably machine dependent. But it must support the nextand previous record operations, which is easy with this implementation. Since therecords may be of different sizes, it is plausible that this scheme is actually used onmost systems. But of course, the encoding and size of the length marks may vary.Appendix C gives a full definition of the format used for the F77 Data Set format.


3.7 Disks, File Systems and Data Set Sizes

Data has been stored using an Extended Logical Volume (XLV) file system onthe SGI Origin 2000. Three SCSI-controllers manage six striped disks. No otherredundancy or additional safety mechanisms are used than the ones used for normalfile systems on single disks. The benefit of this comes from storing and retrievinglarge files. Striping width is in this case six disks and the striping length is 64kilobytes. A file occupies at least the stripe length on all disks, in this case a total64 · 6 = 384 kilobytes. Accessing blocks of these sizes actually makes a parallelrequest to all disks to read the stripe length of data. In theory, if the disks are ableto deliver approximately 12 MB/s per disk, this equals to 72 MB/s from all disks.The bandwidth of the SCSI-channels provide 120 MB/s together. For the currentdata sets used, a maximum of 57.4 MB/s have been achieved reading the volumedata set, see appendix A. The sizes of the data sets are displayed in table 3.3.

Table 3.3: Data Set Sizes

Data Set Slices Slice Size Total SizeHeight Field 15600 27.0 kB 421 MBParticles 15600 1.27 MB 19.8 GBDensity Volume 15600 2.95 MB 46.0 GB

Chapter 4

Visualisation of Data from PICSimulations

Visualisation of data from PIC simulations exposes some general issues concern-ing the analysis of simulation data. In our test scenario the physicist wants tounderstand the behaviour of large groups of particles and how they interact withthe electromagnetic fields. In essence, the overall behaviour of the particles fordifferent situations is the phenomena under study. In this chapter the visual objectsused for visualisation of such properties are described.

4.1 Key Features

A PIC simulation produces a large number of data slices, each representing onetime step. The slices represent snap-shots of the simulation taken at several differ-ent time-steps throughout the simulation. The purpose of the Visualisation System(VS) developed during this diploma work is to provide an interactive animationwith an emphasis on high through-put of simulation data results. The high perfor-mance support from the Data Management System is essential, but it is needed tobuild the visualisation system in conjunction with that. Some of the key featuresprovided are defined as follows:

• Visualisation of particle densities in two and three dimensions. The two-dimensional phase-space uses the(x, vx) values from the particle states. Forthe three-dimensional phase-space we use the(x, vx, vy) values from theparticle states. The Height Field and Density Volume represents this data.

• Visualisation of individual particles in three-dimensional phase-space. Thephase-space values(x, vx, vy) are mapped to Cartesian space(x, y, z). The3D Points visual object represents this data.

• Visualisation of background proton density in the one-dimensional space of(x). This is provided by the Ribbon Graph.

25

26 CHAPTER 4. VISUALISATION OF DATA

doMouse

doMouseEvent

doKey

doSpecialKey

doDraw doDrawStuff

glutMainLoop

Figure 4.1: Visualisation Main Loop and Callbacks

• Interactive view control of the visual objects. The user can rotate, pan, andzoom the objects to find the desired view.

• Animation control by playing forward or backward, stepping individuallyfrom slice to slice, and select fast forward or fast backward with selectablespeed.

4.2 Visualisation System Design

The design of the Visualisation System is based directly on the OpenGL standard[19]. OpenGL stands for Open Graphics Library. It provides general polygon han-dling, lighting, and texturing, etc. The OpenGL GLUT (GL Utility Toolkit) [19]is used to interface to the Windowing System. It provides a platform independentinterface and is very easy to use and get started with. For better control one shoulduse native systems. The basic principle is to assign a set of callback functions tothe GLUT. The GLUT’s main loop then calls these functions as response to variousevents. Figure 4.1 shows the general idea and the functions used in the program.

doMouse This function handles the continuous updates of the mouse pointer po-sition. Depending on the mouse button pressed and combination of modifierkeys (shift, control), different parameters are controlled.

doMouseButton Whenever a mouse button is pressed or released this functionis called. The mouse can thus be used for different purposes depending onthe active modifier key states and mouse button pressed.

doKey The keys are used for many different things, and this function handles allthose, with the exception of the special keys described next.

doSpecialKey Special keys refer to up-key, down-key, function keys, etc. Thisfunction handles such keys.

doDraw This function sets up the camera and lights for the scene to draw. It callsthedoDrawStuff function.

4.3. DATA MAPPING AND TRANSFORMATION 27

doDrawStuff This function does all major rendering of the graphical objects.

The largest function is thedoDrawStuff function. It checks all objects if theyare active and if so asks that object to draw itself. To initiate generation of the nextframe during animation, this function first informs the visual objects to start buildthe next frame so that the next frame can be generated while the current is drawn.The redraw functiondoDraw is always invoked whenever the viewable windowarea needs to be repainted and when an explicit redisplay message is received inthe input stream of window events. Keys and mouse actions, along with animationcontrol posts a redisplay message by calling theglutPostRedisplay functionin the GLUT. Additionally, adoReshape callback is used to handle reshaping ofthe window.

Except from OpenGL and GLUT all software was developed in this project.There are a wide range of available software packages, free and commercial, whichprovide elements of what has been done in this work. The choice not to use any ofthese is that most of them are optimised for static data and not for time dependentdata. It was also uncertain what data through-put existing packages could provide.Furthermore, it was of interest to learn about the fundamental mechanisms forinteractive animation and large data sets. Implementing the methods oneself isa good experience to reach a deep level of understanding. Also implementingthe graphics and not base the visual objects on higher level software than simplepolygon handling, was a good mean to obtain thorough knowledge in graphicsprogramming.

4.3 Data Mapping and Transformation

The data input can be mapped and transformed to generate different visual objects.The larger the data slices get the less computation and processing is possible withina certain time. There are currently two kinds of topological types in the simula-tion data. The first type is structured data and the second type is unstructured data.Structured data means that cells are defined inN dimensions with equal spacing.Unstructured data means that cells are defined inN dimensions with arbitrary po-sitions. The two types are shown in figure 4.2. The benefit of structured data isthat position in the domain’s space can be defined by the set’s cell order. Thus,the coordinates do not need to be given for each cell within the set. Position in thedomain’s space must be included for each cell in the unstructured data.

Electrons and their phase-space position is an example of unstructured data.In the simulations, around two million electrons have been included. This largenumber could not be handled efficiently, so this data was transformed to structureddata in the form of a density volume (a density area for the protons). The phase-space is divided into a number of cells, and the number of particles within a cellis counted. Thus a scalar field is generated, where the value in each cell definethe number of particles in that partial space. The reduction of particle positions todensity fields is done within the simulation program. To save all particle positions


Structured Data Unstructured Data

Figure 4.2: Structured and Unstructured Data

with the time-resolution used would have resulted in a data set of2000000 · 6 ·15600 = 187 GB. Such a large storage area was not available for this work.

The mapping of any three dimensions to the virtual world dimensions(x, y, z)is fairly straightforward and does not require much processing. However, buildinggeometries from individual points in three dimensions can be more or less intricateand computationally intensive. Other dimensions are often mapped to colour andtransparency to visualise higher dimensionalities.

4.4 Visual Objects

The design of the visual objects was influenced by the data streaming concept[12]. In our case this has been applied so that data for an image is requestedfrom the source, i.e. from files stored on local disks, when it is needed. In a pipe-line fashion, the data is transformed or used to build graphical data (geometricprimitives). In the last stage the graphical data is rendered to the image. The dataflow always goes from source to frame and is transformed to visual objects byusing high performance 3D graphics and multi CPU technology. The continuousstreaming of data at interactive frame-rates is the goal.

The time to build up a visual object from a given data slice varies for differentobject types. A Visual Object is a fairly self-contained entity which representsa visual object in the rendered images. For each update of the input data, i.e.changing the viewed data slice, a visual object must perform the following tasks.

1. Load correct input data to view.

2. Generate graphical data from input data, like polygons, colours, textures, etc.Also called geometric primitives.

3. Render graphical data to image.

4.4. VISUAL OBJECTS 29

Interactivity can be defined on many levels. We will discuss interactivity based onthe three tasks Load, Generate, and Render. The strongest sense of interactivityis gained when all tasks can be done within the time for one update of the videoimage. If a display is given image updates at 60 Hz, the graphics application shouldupdate the graphics data with the same rate. The user will then experience about160 – 1

30 seconds latency (input device handling time neglected). We now define thevideo refresh frequence asf with the periodT = 1/f . Given the times for load,generation, and render asTL, TG, andTR we can denote this condition as

TL + TG + TR < kT (4.1)

and call it the Single Stage Pipeline condition. We have used a constantk ∈ Nto denote that the frame-rate can be reduced. If condition 4.1 is satisfied withk = 1 one can not do better without increasing the refresh frequency of the videogeneration. A high refresh frequency is important for fast camera motion and fullscenery. In our case we can settle withf/2 Hz1, that isk = 2. But this is alwaysan act of balancing with priorities, smoother animation transitions or more detailsin the animation.

With a single CPU system, there is not much one can do to improve the situa-tion other than reduce the individual timesTL, TG, andTR. When multiple CPUsare available the work can be divided among the CPUs. Rendering of graphicscan not be done until the graphics are generated. Generation of graphics data cannot be done until the input data is loaded. A scheme minimising latency wouldinvolve pipelining with a resolution higher than the frame-rate. Such a scheme hasnot been investigated in this work. The resolution used in the following discussionis the frame-rate1/kT . If the work is divided between two CPUs, one CPU canrender graphics data and one CPU can perform the load and generation task. Thecondition 4.1 is thus split up into the following

TL + TG < kTTR < kT

(4.2)

and we name this a Two Stage Pipeline, shown at the top in figure 4.3. A conse-quence is that a latency for load and generation of2kT is introduced. The renderinglatency is stillkT . This means that there is still a minimum latency for camera po-sition, object orientation, etc. But we have introduced a higher latency for whatdata to load and control of the graphics data generation.

If more latency is acceptable, a Three Stage Pipeline can be introduced withthe following timing condition.

TL < kTTG < kTTR < kT

(4.3)

1The video refresh-rate is still 60 Hz, but the image is updated with 30 Hz


In figure 4.3 the middle example illustrates equation 4.3. We thus have a minimumlatency of3kT for what to load,2kT for how to generate graphical primitives,andkT for how to view the scene. For example, if we hit the Play Animationkey it takes about3kT to 4kT until we see it started. With a frame-rate of30 Hz(T = 1/60 andk = 2) this is a0.1 – 0.133 seconds delay for animation control(input device delay is neglected).

The bottom illustration in figure 4.3 shows a Two Stage Pipeline which usesdouble rendering. The following condition must be fulfilled for this case.

TL + TG < 2kTTR < kT

(4.4)

One ofTL andTG is greater thankT , but they can both be done under2kT seconds.If the rendering timeTR < kT , the same graphics data can be rendered twice toachieve higher interactivity with respect to camera motion and objects placementand orientation.

In the VS, we have designed and implemented visual objects that use a twostage pipeline. A visual object has a separate load and generation thread whichruns in parallel with the rendering thread (the main thread). It is built so that if therender timeTR is less than(TL+TG)/2 it automatically should render the graphicstwice for each new update of input data. With additional CPUs, multiple objectscan generate graphics data simultaneously.

The file system is an other bottle-neck. The volume and particle visual objectsboth use the disk system with such a high data-rate that both of them can not beused simultaneously (as shown in appendix A) with the present high-end hardwareconfiguration. Rendering of the visual objects remains as the final crucial task.More advanced techniques must be applied to parallelise the rendering part. In ourcase we must share the timekT between all visual objects in the image, such thatthe following condition holds.

TR1 + TR2 + . . . + TRn < kT (4.5)

Where1/kT gives the real frame-rate.The OpenGL standard provides several ways to feed the graphics system with

geometric primitives. One of the most efficient techniques involves the use of ar-rays, in which data for many triangles or quads (polygons with four vertices) arestored. OpenGL then provides the rendering of complete arrays with multiple poly-gons (triangles or quads). Operations likeglDrawArray can be used to renderall that data with a single function call (setting up the arrays takes a few calls). Ver-tex, normal, colour, texture coordinates, and index arrays are managed by a classcalledpdv_PolygonData . For double buffering of geometric primitives, twosuch objects are used for a visual object. One is used to store newly generated dataand the other is used to render the current frame. To increase performance, all datais interleaved within one large array. The improved performance comes from the

4.5. 1D - RIBBON GRAPH 31

frame 0 frame 1 frame 2 frame 3 frame 4 frame 5 frame 6

Two StagePipeline

load/gen render

load/gen render

load/gen render

load/gen render

ThreeStage

Pipeline

load generate render




Two StagePipelineDouble

Rendering

load/generate render render

load/generate render render

load/generate render

Figure 4.3: Load, Generate, and Render Pipeline

reduced scattering of memory accesses. An interleaved array keeps vertex data to-gether and the system does not need to read from multiple locations in the memory.Examples of compositions of interleaved arrays are found in figure 4.4.

4.5 1D - Ribbon Graph

The Ribbon Graph implements the visualisation of a one-dimensional scalar func-tion, i.e. a curve. The curve is defined by a set of points(xi, yi). Using a structureddata set, the values ofxi are implicitly deducted from the position of the valueyi. To draw this line in 3D, we need to add a value for thez-dimension, a fixedvalue z0 is used. To provide a better visual object, this curve is extended to asurface by extrusion in thez-direction. Point pairs are thus created by(xi, yi, z0)and (xi, yi, z0 + ∆z), with the ribbon width defined as∆z. A surface is thencreated by building rectangles from the following points:(xi, yi, z0), (xi, yi, z0 +∆z), (xi+1, yi+1, z0 + ∆z), (xi+1, yi+1, z0). The ribbon is created by drawing allthe quad polygons fori ∈ [0, n− 2] wheren is the number of points in the originalcurve. The valueyi is also set to modify the colour of the corresponding vertex.The ribbon can be rendered both in flat shading and smooth shading mode, ex-plained in section 4.6.1.

The Ribbon Graph is implemented by thepvv_RibbonGraph andpdv_Ribbon classes. The ribbon is constructed from a Quad Strip [19]. The


16 bits vertices, normals, and texture coordinates, 32 bits RGBA

X0 X0 X0 NX0 NY 0 NZ0 R0 G0 B0 A0 TU0 TV 0 X1 Y1. . .

32 bits vertices, 16 bits normals, 32 bits RGBA

X0 Y0 Z0 NX0 NY 0 NZ0 R0 G0 B0 A0 X0. . .

32 bits vertices/points

X0 Y0 Z0 X1 Y1 Z1. . .

Figure 4.4: Interleaved Graphics Data

Figure 4.5: Ribbon Graph

4.6. 2D - HEIGHT FIELD RENDERING 33

Figure 4.6: Height Field

colour coding is computed by a subclass topdv_TransferFunctionIfc .Normals are computed for either flat shading or smooth shading, the principle is asimpler one-dimensional problem and is discussed in detail for the Height Field.Figure 4.5 (D.4 in colour) shows an image of the Ribbon Graph. The size of thisscalar field is approximately a kilobyte in the data slice.

4.6 2D - Height Field Rendering

A Height Field is based on a grey-scale image, a two-dimensional scalar field.The value at a specific point determines the height of the mesh in that point.The Height Field is also coloured by a texture or specific colouring of the ver-tices. This Height Field uses structured points and is implemented by the classespvv_HeightField andpdv_HeightField . A view of the Height Field isshown in figure 4.6 (D.5 in colour). The cell values in the data set from the PICsimulation represent the density of particles in the grid cells/subareas.

The size of the scalar field is about some 20 kilobytes and the resolution isabout90 × 300 points, which results in 53222 triangles. The resolution varieswith the different data sets. The Height Field is constructed with OpenGL TriangleStrips [19] and the height is mapped according to figure 4.7.


Figure 4.7: Height Field Principle

4.6.1 Normal Calculation for Height Field

To shade (colour) the triangles of the height field, their normals must be definedand computed. There are three common methods for scan-line renderer.

Per Polygon means that the shading of all the pixels in a polygon is computedonce and the same shading is applied to all pixels. Flat Shading refers to thismethod. The normal is calculated once per polygon.

Per Vertex means that the shading is computed once for each vertex, the differentvertex shades are then interpolated for all pixels in-between. The most com-mon method refers to Gouraud Shading [6] or Smooth Shading. The normalis calculated once per vertex.

Per Pixel means that the shading is computed per pixel. The normal is interpo-lated, estimated or computed for each pixel. The most common methodbased on this scheme is Phong Shading [6], which interpolates the normalfor each pixel between the vertices and then shades the pixel.

The available hardware has support for both Flat Shading and Smooth Shading(Gouroud), but not per pixel or Phong Shading. Both smooth and flat shading aresupported by the height field class. The mathematics behind the normal calcula-tion is explained next. Phong shading includes computation of specular effects,these effects can be computed in a per vertex manner, but requires a fine mesh ofpolygons to visually pleasant. The effect is best rendered on a per pixel basis.

4.6. 2D - HEIGHT FIELD RENDERING 35

Flat Shading

Flat Shading only requires one normal to be calculated per triangle. The calculationfor the normal~n is thus done in the following fashion, using the three verticesA,B, andC:

pp

p

��1@@

@I��

��

��

A

B

C

~n =−−−−−→(B −A)×−−−−−→(C −B)

The same normal is applied to all vertices of the triangle, i.e.A, B, andC. Allnormals are normalised, i.e.~n/|~n| before they are sent to the graphics system.

Smooth Shading

A smooth surface can be generated by different methods and algorithms. The basicdifference is whether the normal is calculated per vertex or per pixel. The classicalper-vertex method is Gouraud Shading and is common in real-time systems. Themost known per-pixel method is Phong Shading and is currently not widely usedin real-time rendering systems.

Given a two-dimensional structured grid of points, we can apply a second de-gree curve to any three points. Choosing three consecutive points along either thex-direction or they-direction, the derivative for the middle point can be computed,i.e. the curve’s tangent. The normal is then computed by taking the cross productfor the tangentials in both directions for one point. A second degree curve has theequation

ax2 + bx + c = z (4.6)

in the x-direction, with its derivative

dz

dx= 2ax + b (4.7)

Now choosing three consecutive points(x1, z1), (x2, z2), and(x3, z3), as infigure 4.8. These points are structured so we can also define the points as(x2 −∆x, z1), (x2, z2), and(x2 +∆x, z3). To find the constantsa, b, andc for this curvesegment we need to set up an equation system and solve it. We have replacedx2

with x.

a(x−∆x)2 + b(x−∆x) + c = z1

ax2 + bx + c = z2

a(x + ∆x)2 + b(x + ∆x) + c = z3

(4.8)


x1 x2 x3

z1

z2

z3

~nx

~tx

~tx

Figure 4.8: Second Degree Curve

which has the solution

a = z1 − 2z2 + z3

2∆x2

b = −2x(z1 − 2z2 + z3) + ∆x(z1 − z3)2∆x2

c = x2(z1 − 2z2 + z3) + x∆x(z1 − z3) + 2∆x2z2

2∆x2

(4.9)

The derivative 4.7 thus results in the following expression

dz

dx= 2xa + b =

z3 − z1

2∆x(4.10)

In the same fashion we can find the tangent in they-direction for three consec-utive points(y −∆y, z′1), (y, z2), and(y + ∆y, z′3).

dz

dy= 2ya′ + b′ =

z′3 − z′12∆y

(4.11)

We now have the two tangents~tx and~ty

~tx = (∆x, 0,z3 − z1

2)T

~ty = (0,∆y,z′3 − z′1

2)T

and can define the normal~n to the plane(~tx, ~ty).

~n = ~tx × ~ty

Borders are handled a little different. The tangents are defined using only twoneighbouring points, i.e. a linear approximation.

~tx = (∆x, 0, z2 − z1)T

~ty = (0,∆y, z2 − z′1)T

4.7. 3D - VOLUME RENDERING 37

Figure 4.9: Density Volume

4.7 3D - Volume Rendering

Volume rendering is a concept facilitating the viewing of a complete volume ofdata. Not only the surfaces of objects in the volume, but also their interior isrepresented by the data. Cutting out a section of the volume would thus reveal theinterior of objects, which is exactly how the real world around us works. Objectrepresentation for computer graphics is mostly based on surface models, whereonly the surface of the objects are defined. If one wants to look into such objects,there is nothing to see. Volume rendering is a class of various rendering techniquesthat allow the interior of objects to be displayed. Furthermore, there are many realworld objects and phenomena which have no visible surface, as in clouds or fires.

The applications for volume rendering are many. In medicine, for example,one can analyse bone fractures, the blood flow in the brain, or a complete body. Inthis work we have used volume rendering to show the density of electrons withinthe phase-space(x, vx, vy). A bounding volume is set by upper and lower limitsin the phase-space. Cells within this volume is then constructed by dividing thedimensions into desired resolution (90 × 128 × 128). The value within each cellthen represents the number of particles occupying that sub-volume. Figure 4.9(D.6 in colour) shows a screen dump from the volume visual object. The higherthe intensity, the more electrons are there.

Volume rendering can be divided into two major categories; Object Space ori-


ented and Image Space oriented [6]. Object Space oriented rendering generates theimage on the basis of the volume data space. The advantage of this method is thatthe volume can be divided between multiple processors, each only accessing a frac-tion of the volume. They compute their own contribution to the resulting image,and all contributions are then composed together. The disadvantage is that muchof the volume can be hidden by parts closer to the viewer. Those parts were notnecessary to compute and render. Image Space oriented rendering bases the ren-dering on the viewed image, i.e. in screen space. An advantage with this methodis that the image can be divided among multiple processors, but they must be ableto access the whole volume. It also renders only what is really seen by the viewerand nothing else.

There are many proposed algorithms and solutions for volume rendering. Pre-sented here are three examples. Parker et al [13] have proposed an algorithm forinteractive ray tracing for isosurface rendering. An iso surface is generated bydefining the surface where a functionf(x, y, z) = ρ is constant. By using Brick-ing as a memory layout scheme they are taking advantage of cache and page sizesfor use in parallel rendering. Their algorithm has been used for static data whereaswe have a time-varying field. An other proposed algorithm is given by Han-WeiShen et al [14] and suggests the usage of a Time-Space Partition (TSP) Tree. Ittakes into account the coherence in time and space to create a tree structure wherethe data size and the I/O overhead can be reduced. Ray Casting [17] is a commonmethod and is similar to the one used in this system, but differs in the way that itdoes not take advantage of the texturing hardware support, as described next.

Effective volume rendering can be achieved with hardware support for textures[18]. A texture is an image which is placed on polygons. Commonly used andsupported are two dimensional textures. If several images are placed on top of eachother we can create a three dimensional texture. In essence it is a three dimensionalfield. The elements in the volume represents the values of a function of threevariables(x, y, z), or (u, v, w) as often used in texture contexts. The values canbe either scalars or vectors. These elements are called voxels (Volume Elements),in addition to pixels (Picture Elements) or texels (Texture Elements). In volumerendering context one speak of voxels.

In this work an Image Space oriented technique called Volume Slicing has beenused. Volume Slicing uses textures and polygons to sample a volume and renderit. This technique enables the use of hardware support for texturing. The usedtechnique is merely a brute force method which is shown to be very successful.The volume slicing can be achieved in a few different ways.

1. With support only for two-dimensional textures, a volume must be sliced intotexture slices. Depending on the view angle the volume is sliced in differentways. If the view direction is closest to thez-axis, slices are made ofx-yplanes. Each texture slice is then mapped on a polygon that coincide withthe texture slice in orientation and position.

2. With support for three-dimensional textures, a volume can be downloaded

4.7. 3D - VOLUME RENDERING 39

Orthographic Projection Perspective Projection

Figure 4.10: Volume Slicing for Orthographic and Perspective Projection

as such to the graphics system. A set of polygons intersecting the volumeis then sampling the texture. The polygons are oriented so their normalscoincide with the viewing direction. There are two cases:

Orthographic Projection The polygons are all parallel with the viewingplane and placed with increasing depth. See left example in figure4.10. Parallel polygons have been implemented in this work, and isalso used in perspective projection as well. As long as the distance tothe volume is not too small, this works for a normal display but not forimmersive display systems.

Perspective Projection The optimum is concentric spheres with increasingradius. See right example in figure 4.10. These spheres are approxi-mated with polygons, such that the normals at the vertices are not de-viating more than a specific angle from the eye direction.

Support for three dimensional textures gives some benefits compared to two di-mensional textures. The volume does not need to be sliced up and reorganiseddepending on the view angle. The border artifacts are reduced, which are verynoticeable when the slicing polygon normals must deviate largely from the viewangle. Three dimensional textures also allow for trilinear interpolation betweenvoxels. A 2D texture can only interpolate in the slice planes.

The images are created in the following manner. First, the volume is definedas a 3D texture. In our case the volume is a scalar field, where each element rep-resents the number of electrons in that subvolume. We set the scalar values in thevolume to modify the opacity in a polygon by replacing the Alpha component ofthe colour. Then a polygon intersects this texture volume. For each pixel drawnin the polygon, the pixel position maps to a coordinate(u, v, w) in texture space.If the coordinate is in between voxels, a value is computed by trilinear interpo-lation to create smooth changes in the rendered image. Trilinear interpolation isan extension to three dimensions of bilinear interpolation, which is defined in twodimensions (see figure 2.6). The composed image is rendered by drawing inter-secting polygons in a back to front order. A polygon is drawn, and the pixels are


coloured by a colourC = (RP , GP , BP , AP ), whereAP is the alpha componentcomputed from the texture values, i.e the electron density. When the polygon isdrawn, the pixels are blended with the background pixels. This blending is doneso that the colour of the backgroundB = (RB, GB, BB, AB) is blended with thepolygon’s pixel colourC to get the new background colourB′ in the followingway.

B′ = (RB + RP AP , GB + GP AP , BB + BP AP , AB + AP AP )

Then the new colourB′ is set in the frame buffer and becomes the new back-ground value. By iteratively performing this operation for a number of polygons,as explained above, the image of the volume is created. Each polygon samplesthe volume in a depth order, and the number of polygons defines the sampling fre-quency. A good sampling frequency is estimated to be four times the number ofcells along a side of the volume. But this is matter of trade-off between imagequality and rendering speed.

The Sliced Volume is implemented by the classpvv_SlicedVolume andcurrently only supports parallel polygons. When the camera is far enough fromthe rendered volume, a spherical approximation of the polygons is not too far fromparallel polygons. The parallel polygons method is currently used for both ortho-graphic and perspective projection. The set of intersecting polygons are managedby an instance ofpdv_PolygonData . The number of polygons to use is definedin the main part of the program. The polygons are distributed with equal distancebetween them and fully surrounding the possible extent of the texture volume. TheSliced Volume object also uses a dedicated loader thread. While one volume isbeing rendered, the next is loaded by a loader thread.

The main bottleneck in this technique and for this particular case is the fill-rateof the graphics system. For each new polygon, another layer of pixels are drawn.To reduce unnecessary pixel fill, the polygons are clipped to the boundaries of thetexture volume. With a viewable area of512×512 pixels and an orthographic viewwith the volume connecting to the borders, this results in

512 · 512 · n√2≈ 185000n

pixels to fill for n polygons. With ann of 256 we thus have185000 · 256 ≈47.5 millions of pixels per frame. To colour each pixel the pixel colour must beinterpolated by trilinear interpolation. And the total number of drawn pixels persecond is limited by the fill-rate.

4.8 3D - Particle Rendering

The fourth visual object implemented shows individual electrons as points. Toanimate and view the whole set of electrons from the simulations has not beenpossible in real-time. So we have settled for a suitable subset of the electrons by

4.8. 3D - PARTICLE RENDERING 41

Figure 4.11: Particles

saving the particle positions from one computational node. This has resulted inapproximately 200000 electrons. A view of the 3D Points visual object is shownin figure 4.11 (D.7 in colour).

The abstract phase-space(x, vx, vy) is mapped to the Cartesian space(x, y, z).To provide a depth cue, the fog support in OpenGL have been used to fade electronsthat are more far away. Lighting has be turned off to reduce the computational loadsince there is no need for the lighting of points.

As an effort to reduce the size of the data, the phase-space coordinates arestored in 16 bit fixed point, thus, each point requires 6 bytes of storage. The 16bit fixed point decimal numbers uses 7 bits as precision and 9 bits as integer part.This gives a total value range of−256.00 – 255.99, which is fully sufficient in thecurrent case. This format can be treated as 16 bit integers as well. And given a suit-able scale modification to the model matrix, the electrons are correctly positionedin space, with a resolution of 65536 steps. The 3D Points visual object is imple-mented by the classpvv_3DPoints . It uses two buffers for coordinate storage.The back buffer is used by the loader thread that loads the data from disk. The frontbuffer is used by the rendering thread to render the points in OpenGL. The coor-dinate buffers are instances of the classpdv_PolygonData . The loader threadassigns the data slice buffer to the polygon data object and sets the actual and usedsize to the number of electrons, i.e. the number of cells in the slice. Hence, data is


never copied and is passed to the graphics system as a vertex array. All the pointsare rendered with a single call toglDrawArrays via thepdv_PolygonDatamember methoddrawArrays .

Chapter 5

Physical Results andPerformance Measurements

5.1 Interfacing Simulation and Visualisation

A format for storage of large data sets was designed. Both a generic model forthe top level structure and a specific format was implemented. The general for-mat allows for extensions to other access channels than disk. For example sharedmemory, TCP/IP channels, or other. The storage model supports very large sets bysplitting them into several files, a process which is made transparent to the user.The design also allows fast random access of the data by the indexing support. Theconcept of partitioning the data set into slices with a slicing dimension is a key fea-ture enabling real-time animation. Read performance tests show that the data canbe accessed at 57.4 MB/s in forward reading. Reading random slices is achieved at45 MB/s for 3 MB slices. A detailed list from the read tests is found in appendix A.The theoretically lowest maximum for disk read on the used hardware and configu-ration is6× 12 = 72 MB/s, if we do not count latency and OS handling. Six diskswere used, each with a lowest maximum sustained transfer of 12 MB/s accordingto disk specifications. The result fromdiskperf , a disk performance tool, showsthat we are close to such rates. For the random read mode, we match the resultsexactly. The results fromdiskperf is also shown in appendix A.

We have investigated mechanisms to save data from the simulation programand how to store it efficiently on the local disk system. The FORTRAN binaryfile format was examined and used to store the data. A set of C++ classes wasdeveloped to handle the specific binary format. They were built on the genericdata set handling classes which can be used for future extensions to allow for otherformats. Storing binary data from a FORTRAN program is a platform specific task,since there is no standard of the format. An option we have not investigated is theintegration of C functions within the FORTRAN programs. Such integrations arealso system dependent, but probably provides easier interfacing and better controlover the generated output.

43

44 CHAPTER 5. RESULTS AND PERFORMANCE

5.2 Data Set Management and Streaming

Data can be requested in a random access fashion as chunks of data slices. Thedata slicing is a key feature for managing the large amount of simulation data. Thedata is partitioned in order to handle the data efficiently. By using binary storageof the data, the achieved band-width from storage devices can be directly used togenerate graphical data and maintain an interactive frame-rate. In the case of par-ticle rendering (3D Points) and volume rendering (Sliced Volume) the data slicesare directly used as graphical primitives without additional transformations. Withthe increasing sizes of the two-dimensional data slices used by the Height Field,the generation of graphical data becomes more intensive. However, generation ofgraphical data can easily be parallelised in this case. Parallelisation of this task hashowever not been implemented.

The possibility to select animation speed has been used to visually analyse thephase properties of plasma waves. For example, choosing steps of 105 slices in fastforward mode, shows the growth of the wave in the plasma beam where the waveis kept fixed. When the instability sets in, we clearly see that the wave frequencyis changed since the wave centre moves strongly. We also see transformations andthat the electrons are rotating in a magnetic field, in normal animation speed this isnot as evident as in100× speed.

5.3 The Visual Objects

The Ribbon Graph, Height Field, Sliced Volume, and 3D Points together provideda very useful tool for the analysis of simulation data. The data streaming and pipe-line design provide the interactivity we want. By rotating and zooming the visualobjects we obtained a much better sense of the three dimensions, as compared totwo-dimensional/image animations.

The volume rendering performance has been measured in a 512x512 windowwith orthographic projection. The number of slicing polygons has been modulatedto explore how the texture fill scales as a function of number of polygons. Thecalculated number of pixels to fill is 262144 per polygon. Ofn polygons,n/

√2 is

inside the volume when not rotated. At most we have achieved an estimated fill of474 M pixels per second, as shown in table 5.1.

To find out the real fill-rate and the time to download the three- dimensionaltexture, equation 5.1 is defined. It models the rendering as a texture download partand a pixel draw part. The texture download time can contain other elements perframe, but the assumption is that texture downloading is the dominant element.

TF p + TLf = 1 (5.1)

In which TF is the time to fill 1 M pixels.TL is the time to download the texturefor each frame. Finallyp is the million number of pixels filled per second (as incolumn Pix/s) andf is the current frame rate (as in column Frame-rate). Now we

5.3. THE VISUAL OBJECTS 45

Table 5.1: Sliced Volume Rendering Performance

2.81 MB Data Slices -90× 128× 128 16 bit cellsPolygons Size Pix/frame Pix/s Frame-rate Band-width

512 512× 512 95 M 474 M 5 Hz 7.5 MB/s256 512× 512 47 M 356 M 7.5 Hz 20.5 MB/s128 512× 512 24 M 261 M 11 Hz 30.0 MB/s64 512× 512 12 M 154 M 13 Hz 35.1 MB/s32 512× 512 6 M 94 M 15.8 Hz 43.2 MB/s16 512× 512 3 M 47 M 15.8 Hz 43.2 MB/s

1.41 MB Data Slices -90× 128× 128 8 bit cellsPolygons Size Pix/frame Pix/s Frame-rate Band-width

512 512× 512 95 M 474 M 5 Hz 6.0 MB/s256 512× 512 47 M 356 M 7.5 Hz 9.6 MB/s128 512× 512 24 M 261 M 11 Hz 15.0 MB/s64 512× 512 12 M 190 M 16 Hz 21.6 MB/s32 512× 512 6 M 119 M 20 Hz 28.5 MB/s16 512× 512 3 M 62 M 21 Hz 28.8 MB/s

use two rows in the table to solve this equation. We have used the first row togetherwith every other row. This was done for both the 16 bit and 8 bit data set. Theresults, the solution toTF andTL is shown in table 5.2 as1/TF (Fill-rate) andTL

(Load-time).If our simple model (equation 5.1) is valid, this shows that replacing the texture,

as done with withglTexSubImage3DEXT takes a substantial time. The size ofthe texture is not affecting the download times much. The technical specificationof the Onyx2 InfiniteReality specifies that the fill-rate is 624 M textured pixels,800 M trilinear interpolations, and 800 M voxels, all per second. Our estimatedfill-rate shows that we are close to peak performance. The total average is(641 +604)/2 = 622 M pixels/s. Some figures actually exceed peak performance, but thelow precision in the figures and uncertainty of the correct number of filled pixelsintroduces a measurement error.

Rendering 212000 points with Z-buffering is maintained in 30 frames/s. Thesustained disk transfer is in this case 36 MB/s. As shown by the read tests inappendix A, the maximum transfer for the particle data set is 43.5 MB/s. So itis not possible to achieve 60 slices/s, which would require 72 MB/s. We used afactorf ∈ [0, 1] to render a fraction of the points. To further test the performancethe particles are rendered once, twice, and three times, as in the columns Single,Double, and Triple. The result of varying the factorf is shown in table 5.3.

The conclusion made from this is that the number of points could be slightlyincreased to approximately 250 thousand points per slice, and still maintain 30Hz (2 · 212000 · 0.6 = 254400 and30 · 254400 = 7632000 = 7.27 M). On theother hand, reducing the number of particles to 125 thousands per slice shouldresult in 60 frames per second. This is still a visually large amount of particles andgives a good image of the physical process. However, accessing less data per read


Table 5.2: Fill-rate versus Volume download

2.81 MB Data Slices -90× 128× 128 16 bit cellsPolygons Fill-rate Load-time Frame-rate

512 - - 5 Hz256 679 M 0.067 s 7.5 Hz128 622 M 0.055 s 11 Hz64 643 M 0.059 s 13 Hz32 620 M 0.054 s 15.8 Hz16 641 M 0.059 s 15.8 Hz

Avg. 641 M 0.059 s -1.41 MB Data Slices -90× 128× 128 8 bit cellsPolygons Fill-rate Load-time Frame-rate

512 - - 5 Hz256 679 M 0.066 s 7.5 Hz128 622 M 0.055 s 11 Hz64 576 M 0.043 s 16 Hz32 566 M 0.040 s 20 Hz16 575 M 0.043 s 21 Hz

Avg. 604 M 0.049 s -

Table 5.3: 3D Point Rendering Performance

Single Double Triplef FPS Pts/s FPS Pts/s FPS Pts/s

1,00 17 7208000 11,5 73140000,95 13 78546000,90 13 74412000,85 14 75684000,80 15 76320000,75 16 76320000,70 16 71232000,65 31 4271800 22 6063200 16 66144000,60 60 7632000 30 7632000 16,5 62964000,55 60 6996000 31 7229200 22 76956000,45 23 65826000,40 30 76320000,20 60 7632000Average 6299933 7033100 7309108Total average 6880714

5.4. PHYSICAL RESULTS 47

operation reduces the disk band-width performance. It is currently not resolvedwhat performance value in the Onyx2 Technical Specification that applies to pointrendering. The closest interpretation is the number of vectors per second, which isstated to be 7.1 M. A figure which we seem match very good.

The Ribbon Graph is rendered in 60 Hz. The number of polygons in this visualobject is just a few hundreds. The Height Field contains significantly more poly-gons. The resolution in the data sets vary between90 × 150 and90 × 500 whichequals to 26522 – 88822 triangles. For a resolutionx×y the number of triangles is2(x− 1)(y− 1) For each column in the grid a triangle strip is created. These stripscontain2(y − 1) triangles and use2y points. The actual points created is stored inax× y grid and an index table is used to draw each strip. Thus the number of ver-tices in memory is reduced toxy compared to2(x−1)y that is sent to the graphicssystem. An alternative is not to use an index table. It requires twice as much mem-ory, but instead memory locality is increased to a single stream of data, comparedto three streams required when using index tables. A stream here means readingmemory in a straight forward fashion, reading from addressa1, a2, . . . , an−1, an.

The achieved frame-rate for a set with a resolution90 × 300 is 20 Hz whichmeans that we render20 · 53222 ≈ 1.06 · 106 polygons per second. This is a verysmall fraction 1

10 of the maximum as listed in appendix A. The reasons for this canlie in the use of index tables and that the strip length is about 600 triangles large.The index table results in scattered memory accesses. The long triangle strips maybreak down parallelism and fill up FIFOs in the graphics hardware. We have notinvestigated this further in this work.

5.4 Physical Results

From a physical research view, the results have exceeded our expectations. TheVisualisation Toolkit has not only helped confirming our initial assumptions basedon analytical methods, it has also provided us with new ideas, some of which wediscuss below. This is because the physical properties and phenomena and theirinterpretation is strongly emphasised by the animation and the possibility to ma-nipulate the visual objects (rotation, panning, zooming).

5.4.1 Unmagnetised Plasma

The case of an unmagnetised plasma allows us to represent the physics using twodimensions (one spatial componentx and one velocity componentvx). We havechosen this case because it is well understood [1]. This allows us to test the simu-lation and visualisation as well as our understanding of underlying physics. Sincethe physics can be visualised in two dimensions we have used the Height Field.Previous work has assumed that the amplitude of the wave is constant [1]. We havelooked at the more realistic case of time-dependant systems (the wave amplitude isa function of time). In our case the wave grew out of simulations noise levels until


it saturated by forming a trapped particle island, see labelFirst Island in figureD.3. Previous work [11] by Krasovsky has shown that the trapped particle islandis unstable. His analytical approach is, however, only capable of following the ini-tial stage of the instability. Our simulations/visualisation verified not only that hisprediction of the unstable trapped particle island, but also allowed us to follow thenon-linear evolution of this instability. We have shown that the instability saturatesby forming a second trapped particle island, see labelSecond Islandin figure D.3.

The simultaneous presence of two trapped particle islands has now allowed usto link our results with another type of non-linearity, the interaction between twotrapped particle islands. This has, for example, been examined by Escande [5]. Hepredicts that the two trapped particle islands will deform under their mutual influ-ence. We have verified this, since the second island has not the expected ellipticalshape. We are currently examining the consequences of this deformation.

Another prediction by Escande [5] has been the appearance of a chaotic bandbetween the two trapped particle islands. Our visualisation shows that this chaoticband is apparently associated with an increased electron density, see labelSecondBeamin figure D.3. The reason for this increased density is unclear and currentlyunder investigation. In addition we have identified a region within this band (calledThird Island in figure D.3), significantly differing from the band. Future work willinvestigate the character of this region.

A side effect we did not anticipate was what a sensitive tool the VisualisationSystem was in order to detect frequency changes. Any changes in the propagationvelocity of the first island, are related to a change in the phase velocityω/k of theinitial wave. Our simulation box length has fixedk, therefore any change inω/kimplies a change inω, the frequency of the initial wave. The apparent sensitivity ofthis method has been greatly enhanced by tuning the animation such that the waveis almost stationary in the visualisation frame of reference. Any change inω/k canthen be associated with a displacement of the first island.

5.4.2 The Surfatron

Trapped particle islands in magnetic fields have previously been analysed for waveswith constantω, k, and electric field amplitude [10]. The presence of a magneticfield implies that the physics now involves three dimensions(x, vx, vy). This isbecause a magnetic field inz-direction couples the two velocity componentsvx

andvy perpendicularly to it. Karney and Bers have been able to reduce the rele-vant physics to two dimensions by means of a transformation into a more suitablecoordinate system. This coordinate transformation is however only valid for verylimited cases. Our three dimensional visualisation tool allows us to circumvent thistransformation and is therefore not bound by the restrictions in Karney and Bers.

In these simulations we have proven the existence of the surfatron (R. Bing-ham, private communication). An electrostatic wave trapping a charged particleforces this particle to move with the phase velocity of the wave. The magnetic fieldaccelerates the particle perpendicularly to the wave. In the absence of the wave this

5.4. PHYSICAL RESULTS 49

would express itself in the particle gyrating around a magnetic field. Since the waveenforces a motion of the particle with the wave, the particle must accelerate up toarbitrary velocities perpendicularly to both the wave and the magnetic field. Wehave found some mechanisms that prevent the surfatron from accelerating chargedparticles to velocities close to the light velocity. In the six lower images in figureD.9 and images in figure D.12 one clearly sees the acceleration of electrons in the(expected)vy-direction. The mechanism that stops the growth is that the frequencyω of the plasma wave changes and goes out of resonance with the proton beam,thus the energy source is switched off. Therefore, only the energy in the wave canbe used to accelerate the electrons. Nevertheless, we have shown that the surfatronexist and is capable to accelerate electrons to0.5c, see figure D.2. We also see thatthe distribution is not a Maxwellian and the assumption of Maxwellian heating canonly serve as a first approximation.


Chapter 6

Conclusions

In this diploma work I have successfully utilised the Onyx2 RealityMonster forScientific Visualisation. I have explored the data streaming paradigm for high datarates. It has resulted in a real-time animation and visualisation system and, at thesame time, I have produced results in research field of plasma physics. I havealso developed a software frame-work for real-time animation and visualisation oflarge data sets. Four visual objects have been implemented which together providea powerful tool for analysis and interpretation of simulation results.

The capabilities of a graphics system, such as the SGI Onyx2, is important forreal-time properties of a visualisation system. The effective use of disks, process-ing power and graphics resources is essential to provide a complete system. Onlyhigh-end graphics, general processing power, or high-end disk storage and speed isnot enough. But it does not help to have all of these, if one does not have a softwarethat enables the true potential of the system. Within the software frame-work, ex-tensions can easily be made by adding new Visual Objects, Data Set formats, DataAccess channels, etc. The Visual Objects implemented have given us new insightsabout space plasma physics of the phenomena under study. Furthermore, interfac-ing between FORTRAN simulation programs and C++ animation and visualisationprograms has been explored.

Our visualisation program has allowed us to intuitively understand well estab-lished physical processes and simultaneously it revealed shortcomings of the ana-lytical assumptions made. The latter has attracted great attention from the plasmaphysics community and we thus consider the results to be publishable in plasmaphysics journals. We hope that the saturation mechanism for the surfatron can bepublished in the Physical Review Letters, which is the most prestigious journal ofgeneral physics. The results have also formed a core element of the public displayon the open week at the NSC, attracting over 1200 visitors.

51

52 CHAPTER 6. CONCLUSIONS

6.1 Critics

In the work our emphasis was towards performance rather than a fully featuredapplication and to respond to the goals set up for this project. Nonetheless, thevisualisation system currently lacks a user friendly interface. The mouse is usedto rotate, pan, and zoom. It is also used to manipulate the intensity of the SlicedVolume visual object. All other operations are performed by operating the key-board. No labels other than for the coordinate axis are provided. Labels like cur-rent time-step, minimum and maximum values and scales on the axis should be inthe graphical view. User interface in 3D graphics is currently not well investigatedin general. Maximum and minimum values are not contained with the data sliceinformation. Hence, assumptions about suitable scales must currently be built intothe program.

6.2 Future Work

Including labels and scales is needed in order to use the images for publicationpurposes. We also intend to distribute this software package to other scientificvisualisation groups. A better decoupling of the graphics generation and the ren-dering is required in order to use a multi-pipe renderer for Power walls, Sphericalwalls, Cubes, stereo displays, etc. A threaded application, as the current VS donot allow for CPU locking and is not as flexible as the use of separate processes.The adaption to and use of multi-pipe systems is also put on the agenda for comingefforts. Thus, we can take this system into immersive Virtual Reality and step intothe simulations.

We have many ideas for extensions and future work with the current system.As a part of the animation control I would like to include recording of scene ma-nipulation so it can be reused for higher quality rendering, which is too slow forinteractive operation. We would also like to include generation of photo-realisticrendering by exporting graphics to formats that can be rendered by such systems,i.e. RenderMan or POV-Ray.

Worth investigating is the idea of subframe resolution of the load, generation,and rendering pipe-line to fully exploit the streaming paradigm. The goal of thisis to reduce the latency in the system. Currently the resolution is based on theframe-rate1/kT .

To improve the VS as a tool to analyse frequency and velocity changes I wouldlike to be able to interactively select a frame of reference in the time-space domain.We think that this can be done without degrading performance noticeably.

Acknowledgement

The National Supercomputer Centre (NSC) is greatly acknowledged for lettingme use the Onyx2 RealityMonster and for all those gigabytes of storage, and mysincere apologies goes to all other users who had to suffer from low disk space. Iwould also like to thank the people at the NSC for the many cheerful coffee breaks.

Special thanks are due to Niclas Andersson who has given excellent support,especially in technical matters, and his insight in Onyx2 hardware and operatingsystem has been instrumental during the progress of the project.

Mark Dieckmann has provided essential support and helped me to understandplasma physics and write up this report.

Many thanks to my tutor and examiner Anders Ynnerman for his support andwork to ensure a fascinating future for scientific visualisation at Linköpings uni-versitet.

The web site Shoutcast has provided excellent links to Techno music MP3-streams, keeping me concentrated on coding and writing the report.

I also would like to thank my parents for their interest and encouragement inmy education and this project.

Without the fullhearted support from my beloved family this work and all yearsof studies would not have been possible. The love from my wife Jenny (to whomI am married) is a never ending source of joy and inspiration. Her patience duringall those late nights of working and writing is a favour I wish to be able to return.To watch my son Henry making continuous progress in life, his laughter and joy,is a blessing and a source of happy thoughts.

53

54 CHAPTER 6. CONCLUSIONS

Appendix A

Onyx2 RealityMonster -Configuration and Performance

A.1 Configuration

The configuration of the used system is one Origin 2000 rack with 12 R10000250 MHz MIPS CPUs and two Onyx2 racks with three InfiniteReality pipes. Inaddition to normal system disks and file systems, 94 GB of storage was availablein a striped disk configuration of 6 disks. The striping is handled by the OS. Thesystem has a total of 6 GB RAM as a single image distributed shared memory.Each InifiteReality pipe has 4 Raster Managers boards, each having 64 MB oftexture memory and giving a total of 256 MB of texture memory per pipe.

The graphical performance for this type of system is shown in table A.1. Num-bers are taken from the Technical Specification of the SGI Onyx2. Since multi-pipeutilities have not been used, the performance of one fully equiped InfiniteRealitypipe is given.

A.2 Performance

The following is the output fromsar -D 10 2 . The debug programF77BinaryDataSet was used to do sequential forward read on the particle dataset. The data was read with the Data Set classes and read slice per slice. The blocksize, i.e. the size of a slice is 1.21 MB. The disk block-size is 512 bytes. Theaverage then sums to 42.4 MB/s.

IRIX64 lustig 6.5 07151432 IP27 03/14/00

10:50:45 device %busy avque r+w/s blks/s w/s wblks/s avwait avserv10:51:15 dks0d4 77 2.3 126.2 14621 0.0 0 9.8 6.1

dks0d5 73 2.3 123.9 14509 0.0 0 9.1 5.9dks1d4 73 2.2 122.2 14426 0.0 0 9.2 6.0dks1d5 75 2.3 127.4 14675 0.0 0 9.3 5.9dks6d4 73 2.3 124.5 14533 0.0 0 9.0 5.9dks6d5 74 2.3 125.9 14603 0.0 1 9.5 5.9

55

56 APPENDIX A. ONYX2 REALITYMONSTER

Table A.1: Onyx2 InfiniteReality Performance - Single pipe with 4 Raster Man-agers

Operation PerformancePolygons/s 10.7 MPixel fill, smooth, Z 896 MPixel fill, textured, AA, Z 624 MAnti-aliased vectors/s 7.1 MTrilinear interpolations/s 800 MConvolutions/s (5× 5 sep. RGBA) 12.7 MVoxels/s 800 M24-bit floating-point Z yesColor 48-bit RGBAFrame buffer size 320 MB

10:51:45 dks0d4 74 2.3 124.7 14457 0.0 0 9.5 5.9dks0d5 72 2.3 124.5 14444 0.0 0 9.3 5.8dks1d4 71 2.3 122.2 14341 0.0 0 9.0 5.8dks1d5 72 2.3 123.5 14403 0.0 0 9.1 5.8dks6d4 74 2.3 123.6 14405 0.0 0 9.2 6.0dks6d5 74 2.3 123.7 14409 0.0 1 9.6 6.0

Average dks0d4 75 2.3 125.4 14539 0.0 0 9.7 6.0dks0d5 73 2.3 124.2 14477 0.0 0 9.2 5.8dks1d4 72 2.3 122.2 14383 0.0 0 9.1 5.9dks1d5 74 2.3 125.5 14539 0.0 0 9.2 5.9dks6d4 73 2.3 124.0 14469 0.0 0 9.1 5.9dks6d5 74 2.3 124.8 14506 0.0 1 9.6 5.9

The next output is again fromsar , but the data set used for forward sequentialread is the 16 bit volume data set. The slice size is 2.81 MB. The average readperformance sums to 57.4 MB.

IRIX64 lustig 6.5 07151432 IP27 03/14/00





A.2. PERFORMANCE 57

In the following output fromsar . The two data sets where read simultane-ously. The resulting bandwidth from the disks then is summed to 48.4 MB/s. Theaverage of the tests above(57.4 + 42.4)/2 = 49.4 MB/s is close to 48.4 MB/s.

IRIX64 lustig 6.5 07151432 IP27 03/14/00





In the following output I have used the debug and test programF77BinaryDataSet to test random access read of data slices of size 2.7 MB.The performance is lower than for straight forward read, but shows a sustainedband-width of 45.5 MB/s. Output generated fromsar -D . This value is close tothe value fromdiskperf using the same file, that is between 44 – 50 MB/s. Thedisk system was a striped disk system with six disks of 64 kbytes stripe length.

IRIX64 lustig 6.5 07151432 IP27 03/16/00

11:49:40 device %busy avque r+w/s blks/s w/s wblks/s avwait avservAverage dks0d4 85 4.4 125.8 15483 0.0 0 27.9 6.8


The next output log shows the performance when randomly reading data slicesof size 1.2 MB from the six disk striped system. The sustained band-width is 33.8MB/s. This value is compared todiskperf measure of 33.8 MB/s for 1 MBslices.

IRIX64 lustig 6.5 07151432 IP27 03/16/00

11:50:43 device %busy avque r+w/s blks/s w/s wblks/s avwait avservAverage dks0d4 82 2.3 98.7 11562 0.0 0 14.8 8.3



The next four logs are generated bydiskperf . We have compared a stripeddisk system of six disks and striping length of 64 kbytes, a three disk striped systemwhere the length was either 64 kbytes or 512 kbytes. We see that a smaller stripinglength provides higher data rate. Not only for smaller read requests, but also forlarger read requests. Using six disks instead of three disks do not provide thedouble performance, but

The following output shows the performance reading from a standard singledisk system. These measures can work as a base for comparison of striped systems.We will use the measure for random read of 2 MB blocks and forward read of thesame size. A single disk gives us about 14 MB/s for random read and 15 MB/s forforward read. These values are then of performance factor 1.0.

matrix 2# diskperf -D -r512k -m4m Electrons_0.dat#---------------------------------------------------------# Disk Performance Test Results Generated By Diskperf V1.2## Test name : Unspecified# Test date : Thu Mar 16 11:45:22 2000# Test machine : IRIX64 matrix 6.5 07151432 IP27# Test type : XFS data subvolume# Test path : Electrons_0.dat# Request sizes : min=524288 max=4194304# Parameters : direct=1 time=10 scale=1.000 delay=0.000# XFS file size : 1404124800 bytes#---------------------------------------------------------# req_size fwd_wt fwd_rd bwd_wt bwd_rd rnd_wt rnd_rd# (bytes) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s)#---------------------------------------------------------

524288 0.00 15.17 0.00 13.06 0.00 11.871048576 0.00 15.15 0.00 13.38 0.00 13.122097152 0.00 15.16 0.00 14.04 0.00 13.864194304 0.00 15.14 0.00 14.48 0.00 14.28

In the next log we see the performance of a striped disk system of three disks,with a striping length of 512 kbytes. The performance factor compared to a singledisk is 1.84 for random read and 1.97 for forward read.

matrix 1# diskperf -D -r512k -m4m fast/Electrons_0.dat#---------------------------------------------------------# Disk Performance Test Results Generated By Diskperf V1.2## Test name : Unspecified# Test date : Thu Mar 16 13:38:51 2000# Test machine : IRIX64 matrix 6.5 07151432 IP27# Test type : XFS striped data subvolume# Test path : fast/Electrons_0.dat# Disk striping : group=3 unit=1024# Request sizes : min=524288 max=4194304# Parameters : direct=1 time=10 scale=1.000 delay=0.000# XFS file size : 1404124800 bytes#---------------------------------------------------------# req_size fwd_wt fwd_rd bwd_wt bwd_rd rnd_wt rnd_rd# (bytes) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s)#---------------------------------------------------------

524288 0.00 19.61 0.00 15.70 0.00 15.131048576 0.00 29.14 0.00 25.14 0.00 24.04

A.2. PERFORMANCE 59

2097152 0.00 29.86 0.00 25.58 0.00 25.514194304 0.00 33.47 0.00 30.77 0.00 30.60

The following output is fromdiskperf reading the volume data on a stripeddisk system with three disks, and the stripe length 64 kbytes. The same data asused in the previous test. The performance factor is 2.10 for random read and 2.12for forward read. This is slightly better than for 512 kbytes stripe length.

matrix 1# diskperf -D -r256k -m4m fast/Electrons_0.dat#---------------------------------------------------------# Disk Performance Test Results Generated By Diskperf V1.2## Test name : Unspecified# Test date : Thu Mar 16 14:43:43 2000# Test machine : IRIX64 matrix 6.5 07151432 IP27# Test type : XFS striped data subvolume# Test path : fast/Electrons_0.dat# Disk striping : group=3 unit=128# Request sizes : min=262144 max=4194304# Parameters : direct=1 time=10 scale=1.000 delay=0.000# XFS file size : 1404124800 bytes#---------------------------------------------------------# req_size fwd_wt fwd_rd bwd_wt bwd_rd rnd_wt rnd_rd# (bytes) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s)#---------------------------------------------------------

262144 0.00 31.24 0.00 18.63 0.00 16.58524288 0.00 31.73 0.00 23.67 0.00 22.47

1048576 0.00 32.10 0.00 27.33 0.00 26.452097152 0.00 32.10 0.00 29.97 0.00 29.154194304 0.00 32.24 0.00 30.99 0.00 30.62

The output below shows the result from a six disk striped system with 64 kbytesstripe length. We have a performance factor of 3.19 for random read and 4.13 forforward read.

lustig 2# diskperf -D -r64k -m4m Electrons3D_0.dat#---------------------------------------------------------# Disk Performance Test Results Generated By Diskperf V1.2## Test name : Unspecified# Test date : Thu Mar 16 11:24:41 2000# Test machine : IRIX64 lustig 6.5 07151432 IP27# Test type : XFS striped data subvolume# Test path : /scratch/plg_twodem/run-2000-01-03/Electrons3D_0.dat# Disk striping : group=6 unit=128# Request sizes : min=65536 max=4194304# Parameters : direct=1 time=10 scale=1.000 delay=0.000# XFS file size : 46006396800 bytes#---------------------------------------------------------# req_size fwd_wt fwd_rd bwd_wt bwd_rd rnd_wt rnd_rd# (bytes) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s) (MB/s)#---------------------------------------------------------

65536 0.00 20.98 0.00 5.61 0.00 4.14131072 0.00 38.62 0.00 9.39 0.00 7.38262144 0.00 47.81 0.00 17.28 0.00 13.43524288 0.00 60.55 0.00 25.41 0.00 22.07

1048576 0.00 65.45 0.00 34.57 0.00 33.812097152 0.00 62.62 0.00 41.00 0.00 44.184194304 0.00 62.27 0.00 45.02 0.00 50.69


Appendix B

Source Files

B.1 Source Code Organisation

The source is organised into a set of libraries and an application. The librariesare called Patrics Common Library (PCL), and Patric’s Data Visualiser (PDV).Then the application source is kept in a directory called Patric’s Volume Visualiser(PVV). Following is a list of source files and their purpose. For the C++ files, eachclass have their own header and implementation file.

B.1.1 Patric’s Common Library

pcl_types.h contains common definitions and types.

pcl_Mutex.h,C defines the classpcl_Mutex which is an encapsulation of aPOSIX mutex lock.

pcl_Condition.h,C defines the classpcl_Condition . It implements a stan-dard POSIX condition variable.

pcl_Thread.h,C contains the definition ofpcl_Thread which implements han-dling of a POSIX thread.

B.1.2 Patric’s Data Visualiser

pdv_types.h contains common definitions and types.

pdv_CellElementConverter.h,C is a class that converts the cell element data intovarious system formats (float, int, etc.).

pdv_ColorFunctionLinear4.h,C defines a simple transfer function that interpo-lates between two vectors of size 4.

pdv_CoordinateToAddressIfc.h,C defines the interface for coordinate to addresstranslation in data slices.

61

62 APPENDIX B. SOURCE FILES

pdv_CoordToAddr*.h,C is a set of classes that implement different coordinateto address translation schemes.

pdv_DataSet.h,C is the base class for Data Set.

pdv_DataSlice.h,C is the base class for Data Slices.

pdv_F77BinaryDataSet.h,C implements the Data Set type F77 Binary.

pdv_F77BinaryDataSlice.h,C Implement the Data Slice type F77 Binary.

pdv_PolygonData.h,C implements a class to handle polygon data, supports ver-tex, colour, normal, and texture coordinate data.

pdv_HeightField.h,C implements the height field visual object.

pdv_LinearSegmentFunc4.h is a variable transfer function that implements multi-segment linear interpolation.

pdv_Ribbon.h,C implements the visual object Ribbon Graph.

pdv_TransferFunctionIfc.h,C defines the transfer function interface.

B.1.3 Patric’s Volume Visualiser

pvv_types.h contains common definitions and types.

main.C is the main program.

pvv_3DPoints.h,C is the application part of the 3D point visual object.

pvv_ColorFunc1.h,C defines a simple colouring transfer function with only twocolours to interpolate between.

pvv_ColorFuncVar.h,C defines a variable colouring transfer function that takesany number of colours to interpolate between.

pvv_HeightField.h,C is the application part of the height field visual object.

pvv_Lights.h,C contains the setup for the lighting.

pvv_RibbonGraph.h,C is the application part of the ribbon graph visual object.

pvv_SlicedVolume.h,C implements the sliced volume visual object.

pvv_intro.h,c contains the code for the splash screen animation.

pvv_logo.h,c contains OpenGL code for drawing the PVV logo.

CoordinateAxes.h,c contains OpenGL code for drawing the coordinate axes.

B.1. SOURCE CODE ORGANISATION 63

letters.h,c contains OpenGL code for drawing the axis letters.

p3dmath.h,c implements some basic 3D vector routines, like the cross productand normalisation.

64 APPENDIX B. SOURCE FILES

Appendix C

FORTRAN 77 Binary Data SetFile Format

The format of the FORTRAN 77 binary data set consists of one index file andone or more data files. The index file contains information that is general for allslices and information about each individual slice. The common information isheld in the Index File Header and slice information is held in the Index Record.The file begins with the Index File Header and then continues with a number ofIndex Records. There is one Index Record for each Data Slice in the set. Due toFORTRAN specific features, an Index File Header, Index Record, and Data Slicesare embraced within record size. For the index file the prefixed size is included inthe format specified below. But there is also a trailing size marker. A FORTRANrecord is defined as

fortran-record → 〈 content-size, content, content-size 〉

wherecontent-size is the size in bytes ofcontent. It is stored as a 32 bit integervalue. Pointers and offsets in the structures below always refer to the location ofthe record and not thecontent. The index file, denotedindex-file, is definedwith the structure as defined below.

index-file → 〈header-rec, index-recs〉index-recs → 〈index-rec0, index-rec1, . . . ,+recn−1〉

The recordsheader-rec andindex-rec is defined in the next two sections.

C.1 Header

Theheader-rec is a FORTRAN binary record and enclosed within size markers.The table below show the structure of the record content. The structure of the fieldCellComposition is shown in table C.2, this field is repeated for each cell elemenentin the set.

65

66 APPENDIX C. FORTRAN 77 BINARY DATA SET FILE FORMAT

Table C.1: Index File Header

Field Type Size DescriptionMagicID uint 4 The magic ID identifying this formatting, set to

0xDABEDEC4IndexOffset uint 4 Index file offset for the first index recordRecordSize uint 4 Specifies the size of an index record, including size

markersDimensions uint 4 The number of dimensions used by the data slicesFileCount uint 4 Number of files used to store the dataSliceCount uint 4 Expected number of slices of the data setSlicingType uint 4 The binary type used to store slice-dimension val-

uesStorageScheme uint 4 Specifies the data slice storage scheme, see Storage

Schemes belowCellSize uint 4 The size of the individual cells in the data slicesCellElementCount uint 4 The number of elements in each cellCellComposition cellcomp var The cell composition - describes the type for each

element in the cell

Table C.2: Cell Composition Structure

Field Type Size DescriptionElementOffset uint 4 The offset to the element in the cellElementType uint 4 The type of the element

C.2. SLICE RECORDS 67

Table C.3: Index Record Structure

Field Type Size DescriptionFileNumber uint 4 The file number for this data sliceFileOffset uint 8 The file offset for the data sliceSliceSize uint 4 The size of the data sliceSliceValue var 16 The current slice valueDimensionSize uint 4 The resolution/size of each dimension

C.2 Slice Records

The slice record is a FORTRAN binary record and the content is enclosed withinsize marks. The table C.3 show the structure of the content. The field Dimen-sionSize is repeated for each dimension in the slice. Thus each slice can have adifferent resolution. This support the change of resolution whenever needed toprovide detailed information about the data slice.

68 APPENDIX C. FORTRAN 77 BINARY DATA SET FILE FORMAT

Appendix D

Colour Images

This appendix contains a set of colour images.

69

70 APPENDIX D. COLOUR IMAGES

Figure D.1: Super Nova Remnant SN1006

71

1

2

3

4

5

6

−0.4 −0.2 0 0.2 0.40

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Velocity v/c

Tim

e in

ele

ctro

n cy

clot

ron

perio

ds

Figure D.2: Electron Velocities


Second Beam

First Island

Second Island

Third Island

Figure D.3: Second Beam and Empty Island

73

Figure D.4: Ribbon Graph


Figure D.5: Height Field

75

Figure D.6: Density Volume


Figure D.7: Particles

77

Figure D.8: Particles and Height-Field Serie 1



79



Figure D.11: Volume Images Serie 1

81

Figure D.12: Volume Images Serie 2


Bibliography

[1] Wolfgan Baumjohann and Rudolf A. Treumann.Advanced Space PlasmaPhysics. Imperial College Press, London, England, 1996.

[2] Wolfgan Baumjohann and Rudolf A. Treumann.Basic Space Plasma Physics.Imperial College Press, London, England, 1996.

[3] George Coulouris, Jean Dollimore, and Tim Kindberg.Distributed Systems.Addison-Wesly Publishing Company, 1995.

[4] Paul Devine.PhD Thesis. PhD thesis, University of Sussex, U.K., 1995.

[5] D. F. Escande. Large-scale stochasticity in hamiltonian systems.PhysicaScripta, T2/1:126–141, 1982.

[6] James D. Foley, Andries van Dam, Steven K.Feiner, and John F. Hughes.Computer Graphics, Principles and Practice. Addison-Wesly PublishingCompany, Inc., second edition, 1997.

[7] A. A. Galeev. Acceleration of electrons to ultrarelativistic energies byshock waves and synchrotron radiation of these electrons.Sov. Phys. JETP,59(5):965–971, 1984.

[8] Gösta H. Granlund and Hans Knutsson.Signal Processing for ComputerVision. Kluwer Academic Publishers, Dordrecht, The Netherlands, 1995.

[9] John E. Hopcroft and Jeffrey D. Ullman.Introduction to Automata Theory,Languages, and Computation. Addison-Wesly Publishing Company, 1979.

[10] C. F. F. Karney and A. Bers. Stochastic ion heating by a perpendicularly prop-agating electrostatic wave.Physical Review Letters, 39(9):550–554, 1976.

[11] V. L. Krasovsky. Classification of trapped particle sideband instabilityregimes.Physica Scripta, 49:489–493, 1994.

[12] C. Charles Law, William J. Schroeder, Kenneth M. Martin, and JoshuaTemkim. A multi-threaded streaming pipeline architecture for large struc-tured data sets. Course Notes for SIGGRAPH ’99, 1999.

83

84 BIBLIOGRAPHY

[13] Steven Parker, Michael Parker, Yarden Livnat, Peter-Pike Sloan, and CharlesHansen. Interactive ray tracing for volume visualization.IEEE Transactionson Visualization and Computer Graphics, 5(3):238–250, July-September1999.

[14] Han-Wei Shen, Ling-Jen Chiang, and Kwan-Liu Ma. A fast volume render-ing algorithm for time-varying fields using a time-space partition (tsp) tree.Course Notes for SIGGRAPH ’99, 1999.

[15] Abraham Silberschatz and Peter Baer Galvin.Operating Systems Concepts.Addison Wesley Longman, Inc., 5 edition, 1998.

[16] John Vince. Virtual Reality Systems. Addison-Wesly Publishing Company,Inc., 1995.

[17] Alan Watt and Mark Watt.Advanced Animation and Rendering Techniques -Theory and Practice. Addison-Wesly Publishing Company, 1992.

[18] Rüdiger Westermann and Thomas Ertl. Efficiently using grahpics hardwarein volume rendering applications. Computer Graphics Group.

[19] Mason Woo, Jackie Neider, and Tom Davis.OpenGL Programming Guide.Addison-Wesly Publishing Company, Inc., 1997.

Documents

Visualization of Particle In Cell Simulations19671/FULLTEXT01.pdfVisualization of Particle In Cell Simulations Författare Author Patric Ljung Sammanfattning Abstract A numerical simulation