Vis ComputDOI 10.1007/s00371-008-0309-x

O R I G I NA L A RT I C L E

Visual simulation of thermal fluid dynamics in a pressurized waterreactor

Zhe Fan · Yu-Chuan Kuo · Ye Zhao · Feng Qiu ·Arie Kaufman · William Arcieri

© Springer-Verlag 2008

Abstract We present a simulation and visualization sys-tem for a critical application—analysis of the thermal fluiddynamics inside a pressurized water reactor of a nuclearpower plant when cold water is injected into the reactor ves-sel. We employ a hybrid thermal lattice Boltzmann method(HTLBM), which has the advantages of ease of paralleliza-tion and ease of handling complex simulation boundaries.For efficient computation and storage of the irregular-shapedsimulation domain, we classify the domain into nonemptyand empty cells and apply a novel packing technique to or-ganize the nonempty cells. This method is implemented on aGPU cluster for acceleration. We demonstrate the formationof cold-water plumes in the reactor vessel. A set of inter-active visualization tools, such as side-view slices, 3D vol-ume rendering, thermal layers rendering, and panorama ren-

Z. Fan (�) · Y.-C. Kuo · F. Qiu · A. KaufmanCenter for Visual Computing and Department of ComputerScience, Stony Brook University, Stony Brook, NY 11794-4400,USAe-mail: fzhe@cs.sunysb.edu

Y.-C. Kuoe-mail: yukuo@cs.sunysb.edu

F. Qiue-mail: qfeng@cs.sunysb.edu

A. Kaufmane-mail: ari@cs.sunysb.edu

Y. ZhaoDepartment of Computer Science, Kent State University, Kent,OH 44242-0001, USAe-mail: zhao@cs.kent.edu

W. ArcieriISL Inc., Rockville, MD 20852, USAe-mail: billa@islinc.com

dering, are provided to collectively visualize the structureand dynamics of the temperature field in the vessel. To thebest of our knowledge, this is the first system that combines3D simulation and visualization for analyzing thermal shockrisk in a pressurized water reactor.

Keywords Power plant · Pressurized thermal shock ·Thermal fluid dynamics · Lattice Boltzmann method

1 Introduction

The assessment of pressurized thermal shock (PTS) risk inreactor vessels has been of great interest to the designersand constructors of pressurized water reactors (PWRs) innuclear power plants. The purpose of the PTS investiga-tions [8] is to assess whether the brittle fracture of the reac-tor vessel is credible during various off-normal or transientevents. A concern recently raised is the possibility of reac-tor vessel failure due to the formation of cold water plumeswhen cooling the reactor system at an excessive rate duringthese events.

We have worked closely with the PWR scientific engi-neers to develop a computation and visualization system toaddress this issue. We perform computational fluid dynam-ics (CFD) analysis to address this application using a hybridthermal lattice-Boltzmann method (HTLBM) [16]. We sim-ulate the primary system coolant loops inside a reactor ves-sel to determine whether cold water plumes will form in thevessel downcomer due to the injection of coolant in an emer-gency. To improve the simulation performance, we acceler-ate the computation on a graphics processing unit (GPU)cluster, which can handle the remarkably large simulationdomain with reasonable running time. We further divide theirregular-shaped simulation domain of the PWR into non-empty and empty cells and apply a packing technique to

Z. Fan et al.

organize the nonempty cells to reduce memory consump-tion and computation time. We also provide interactive vi-sualization tools to effectively and efficiently convey the in-formation of the temperature field structure and dynamicsto the designers and analysts. To the best of our knowl-edge, we are the first to implement combined 3D lattice-Boltzmann simulation and 3D visualization for PTS analy-sis. We are also the first to use the lattice-Boltzmann method(LBM) in PTS analysis with fast GPU cluster computa-tion.

In previous work, the U.S. Nuclear Regulatory Commis-sion (NRC) has undertaken studies to investigate the per-formance of the reactor vessel during off-normal or tran-sient events. Li and Modarres [17] have discussed in de-tail the history of PTS studies. The PTS investigation re-quires the analysis of a large number of transient events todetermine the thermal hydraulic conditions in the reactorvessel over a period of time. This thermal-hydraulic analy-sis has been performed with the RELAP5 computer code.Various reviewers of the PTS analysis have raised the is-sue of the possibility of cold water plume formation in thereactor vessel downcomer and the potential impact on ves-sel performance. Because of this possibility, it is neces-sary to further use 3D simulation to study whether the fluidcirculation and mixing is sufficient to dissipate cold waterplumes that may be formed. Martin and Bellet [21] have an-alyzed the qualification of finite-element-based and finite-difference-based CFD software for 3D simulation of PTSanalysis.

In these PTS studies, the simulation results are visualizedwith the temperature history plots for several key positionsand the surface color maps. With these elementary visualiza-tion tools, it is difficult for the designers and analysts to in-fer the structure and dynamics of the temperature field. Oursystem provides a set of interactive 3D visualization toolsto help understanding the thermal fluid dynamics and ana-lyzing the structure and dynamics of the temperature field.One of the tools, the panorama rendering (see Sect. 5.3), istailored for the special geometry of the PWR vessel. Oursimulation uses the HTLBM, which is an extension of thelattice-Boltzmann method (LBM) [23]. It models the fluidat a mesoscopic level and hence have the following advan-tages. The boundary condition for the complex PWR ves-sel geometry can be easily handled. The computation opera-tions are local and linear, hence can be fully parallelized onthe GPU and the GPU cluster.

The LBM has been introduced to the graphics and vi-sualization community by Wei et al. [26, 27]. Thurey andRuede [24] have developed free surface fluid simulationswith the LBM. Chu and Tai [3] have used the 2D LBM tosimulate ink dispersion in absorbent paper. Zhu et al. [30]have introduced a two-fluid LBM for simulating mixtures ofmiscible fluids. Zhao et al. [29] have used the HTLBM for

modeling heat shimmering and mirage. Zhao [28] has pro-posed an LBM-based PDE solver. Li et al. [18, 19] have im-plemented the LBM on the GPU, and Fan et al. [10, 11] haveimplemented the LBM on the GPU cluster. Interested read-ers are referred to [22] for a survey of general-purpose com-putation on GPU (GPGPU) and to [1, 4, 12, 13, 15] for otherGPU-accelerated fluid dynamics methods. Our LBM simu-lation is new and challenging in that we simulate the thermalfluid dynamics within the special geometry of the irregular-shaped PWR. If we straightforwardly use a rectangular sim-ulation domain which contains the geometry, only 5.8% ofthe lattice cells are useful. Hence, we propose an LBM im-plementation of irregular-shaped simulation domain on theGPU cluster (see Sect. 4.3) which packs only the nonemptyLBM cells in GPU texture memories.

2 Background

Figure 1 presents the general arrangement of a PWR with aCombustion Engineering plant design. The normal flow pathis from the reactor vessel through the hot legs, the primaryside of the steam generators, and back to the vessel throughthe reactor coolant pumps and cold legs. Plant designs fromother PWR vendors differ in the number of steam generatorloops and the steam generator design among other details.Figure 2 shows the internal structure of a typical reactor ves-sel. The downcomer is the annular narrow region (shown ingreen) between the core support barrel and the reactor vesselwall. The region of interest for this analysis is the cold legpiping and the reactor vessel downcomer.

One class of transients that is of interest to PTS riskanalysis is the small-break loss of coolant accidents(SBLOCA). The LBM analysis focuses on a 5.08 cm (2.0 in)diameter pipe break in the surge line (see Fig. 1) that con-nects to the hot leg. When the break occurs, the reactor

Fig. 1 Typical layout of a combustion engineering PWR. The figureis adapted from [20]. The reactor vessel, cold legs, and hot legs areshown in colors

Visual simulation of thermal fluid dynamics in a pressurized water reactor

Fig. 2 Internals of a typical combustion engineering PWR reactor ves-sel. The figure is adapted from [20]. The downcomer region is shownin green

coolant system (RCS) pressure decreases rapidly from thenormal operating value of 15.5 MPa (2250 psia) at first, andthen more slowly as water-to-steam flashing occurs withinthe RCS.

The RCS depressurization leads to initiation of flow fromthe injection systems typically at a temperature of 305 K(90◦F), which are designed to maintain water inventory inthe reactor system during an LOCA. The decreasing RCSpressure ultimately results in the operator turning off the re-actor coolant pump in each cold leg to prevent pump failure.With no pumped flow in the system, a transition from forcedto natural circulation behavior occurs for a short time. Thisnatural circulation flow keeps the primary system fluid wellmixed (i.e., no cold plumes form in the downcomer). How-ever, continuing loss of RCS fluid inventory stops coolantloop natural circulation flow through both loops. The loss ofnatural circulation flow is referred to as “loop flow stagna-tion” and is a significant point of interest in the PTS analysisbecause of the influence of the cold injection water on thedowncomer is the greatest. During the loop flow stagnationperiod, the fluid flow velocity around the loop, and partic-ularly in the cold leg, becomes practically zero. The onlymass flow through the cold leg during loop flow stagnation isthat from the high-pressure injection (HPI) system, and thisflow is from the injection nozzle (see Fig. 1) downstream ofthe pumps towards the downcomer. The temperature of thefluid in the reactor system at this time is 464 K (375◦F) at apressure of 4.34 MPa (630 psia).

When cold water (305 K (90◦F)) is injected from theHPI nozzles, it flows through the cold leg piping into thedowncomer. Depending on the amount of mixing, thermalstratification may occur in the cold leg, and downcomer andprovide a potential for the formation of thermal plumes. Ifthermal plumes form, a large thermal gradient through the

Fig. 3 The D3Q13 latticegeometry. The particledistribution fi is associated withthe link corresponding to the ei

velocity vector from a latticesite (blue) to its neighbors (red)

vessel wall (relative to the average downcomer fluid tem-perature) can exist that may affect vessel performance. TheLBM analysis focuses on determining whether cold waterplumes form in the reactor vessel when the loop flow is stag-nant.

3 Modeling of thermal fluid dynamics

3.1 Lattice-Boltzmann method

The fundamental idea of the LBM is to model fluid dynam-ics based on the collective kinetic dynamics of many meso-scopic particles. The kinetic equations provide many advan-tages, including easy implementation of boundary condi-tions and use of fully parallel algorithms. The Navier–Stokesequations can be deduced from the LBM, which can beshown by a Chapman–Enskog expansion [2]. See Succi [23]for complete description and discussion of the LBM.

Figure 3 shows a D3Q13 lattice cell (3 dimensions and 13links including the self-link of the lattice site). The variablesassociated with each lattice site are the particle distributionsfi representing the probability of the presence of a fluidparticle with a given velocity vector ei , where i ∈ 0 . . .12represents one of the 13 discrete directions. Particles streamsynchronously along the lattice links to neighboring sites indiscrete time steps. Between streaming steps, they undergocollision, which occurs when particles arriving at a latticesite interact and change their velocity directions accordingto the scattering rules. The discrete kinetic equations are:

fi(r, t∗) = fi(r, t) − 1

τ

(fi(r, t) − f

eqi (ρ,u)

)

⇒ collision, (1)

fi(r + ei , t + 1) = fi(r, t∗) ⇒ streaming, (2)

where r and r + ei locate a lattice site and its neighboringsite along link i, respectively. The distribution denoted asf

eqi represents a local equilibrium distribution, whose value

is usually defined as a linear function (BGK model [23]) ofthe conserved quantities: mass ρ = ∑

i fi and the velocityu = ∑

i fiei . The constant τ represents the relaxation timethat determines the viscosity of the flow. For the kinematicshear viscosity ν, τ is defined as τ = 6ν+1

2 .Only one parameter, τ , is used to control the collision

of particle distribution functions. Therefore this primary

Z. Fan et al.

LBM is called single-relaxation-time LBM (SRTLBM). TheSRTLBM is prone to unstable numerical computation whenused for highly turbulent fluids or incorporated with temper-atures or body forces. Therefore, we use a new version ofthe LBM, multiple-relaxation-time LBM (MRTLBM) [5].It differs from the SRTLBM in that a new collision opera-tor replaces the single-relaxation-time collision in (1). Thenew collision operates in the space of hydrodynamic mo-ments representing density, momentum, energy, etc. Thesemoments, denoted as mi , are mapped from the particle dis-tributions as

|m〉 = M|f 〉, (3)

where |f 〉 = (f0, f1, . . . , fn)T , |m〉 = (m0,m1, . . . ,mn)

T ,and n is the number of links of a lattice site to its neigh-bors. For the D3Q13 lattice, each of the 13 moments hasa physical meaning: m0 is the mass density; m1,2,3 are thecomponents of the momentum vector; m4 is the energy; andthe other higher-order moments are components of the stresstensor and other high-order tensors. Although the values ofthe distributions and the moments vary over the lattice sites,M is simply a constant matrix for a given lattice structure.Mathematically, the MRTLBM collision is described as∣∣f (r, t∗)

⟩ = ∣∣f (r, t)⟩ − M−1S

[∣∣m(r, t)⟩ − ∣∣meq(r, t)

⟩], (4)

where S is a diagonal matrix. Its diagonal elements {si |i =0,1, . . . , n} are the relaxation rates [5]. It has been proventhat the MRTLBM increases the numerical stability [16].Body forces and heat effects can be easily incorporated intothe moments mi , because the moments have explicit physi-cal meanings, such as momentum and energy.

3.2 Hybrid thermal lattice-Boltzmann method

Based on the MRTLBM, Lallemand and Luo [16] havedeveloped the HTLBM for coupling thermal effects tothe LBM. The temperature evolution is governed by thediffusion–advection equation

∂tT + u · ∇T = κ�T, (5)

where κ is the thermal diffusivity of the material, and u is thevelocity. This equation is solved using the finite-differencemethod, where the x, y, and z components of the gradient∇ and the Laplacian operator � are computed by the finite-difference operators [5].

The temperature is coupled to the LBM in order to modelthe interaction between the heat and the fluid dynamics. Themoment m4 represents the energy; therefore, the tempera-ture can be added to the LBM when computing the equilib-rium m

eq4 :

meq4 = n1ρ + n2ρ

2(u · u) + n3T , (6)

where the parameters, n1 to n3, are constants and physicallydefined by the linear analysis. After coupling T to m

eq4 , the

method can model the thermal fluid dynamics.Because the LBM lattice not only discretizes the simula-

tion domain space but also subdivides the angular space (seeFig. 3), the complex surface of the irregular-shaped vesselcan be effectively captured by the intersection points of thelattice links with the surface. Boundary conditions are han-dled with simple local rules after the streaming step. Suchboundary conditions are treated with a bounce back rule. Inthis case, the outgoing particle distribution re-enters the gridat the same lattice site, but associated with the opposite ve-locity. The rule is to copy the mirror image of the particledistributions of the lattice sites that on the fluid side of theboundary to those on the solid side. In the PTS analysis, theplume behavior is affected by the temperature of the ves-sel wall. We include in the simulation the heat transfer fromthe wall to the fluid. Initially, the vessel wall and the fluidare at the same temperature. As cold water is injected, heatexchange occurs between the wall and the fluid.

Typical LBM computation achieves second-order accu-racy for solving the Navier–Stokes equations [23]. The ther-mal LBM method we used has been applied and validated tosimulate turbulent convective flows in 2D and 3D cases [25].Validation studies have been presented and compared tobenchmark data for laminar and turbulent natural convec-tion in a cavity. They show that the scheme yields quadraticconvergence rate in space. Results obtained for both the 2Dand 3D case agree very well with existing benchmark data.

4 Simulation

4.1 Configuration

Figure 4 shows a 3D polygonal model created according tothe specification of the vessel downcomer. The model is cre-ated with Maya Complete 6.0. It includes the downcomer,

Fig. 4 The geometric model of the vessel created for the simulation

Visual simulation of thermal fluid dynamics in a pressurized water reactor

cold legs, and HPI injection pipes. We have carefully con-structed this model based on the layout of the reactor vessel(Fig. 1). The cold water is injected from the HPI injectionpipes and flows through the cold legs into the lower plenum.

To use the LBM as a fluid solver, one has to determinethe LBM configuration of grid size, time step, characteristicspeed and size, etc., observing the configuration rules [27].The following configuration is used in our simulation. Weset the grid spacing (spacing between two neighbor latticesites) δx

.= 0.02 m, and each simulation step δt.= 0.01 s.

The spacing and time step size determine the global latticedimensions of the entire simulation domain at 800 × 600 ×480.

The flow speed of the HPI injection is computed by thegiven flow rate, which is 35.0 lbm/s= 15.876 kg/s. The ra-dius R of the HPI injection pipe is R = 0.27 m and the den-sity of the water is ρ

.= 1 × 103 kg/m3. The flow rate of thewater can be computed as ρπR2U ; therefore, the flow speedof the HPI injection is U = 0.28 m/s. In the LBM simula-tion, the flow speed of the HPI injection is U = 0.14 in LBMunits, which is computed from δx and δt . The viscosity is setas ν = 4×10−3 in LBM units; therefore, the Reynolds num-ber (which is the ratio of inertial forces to viscous forces) inthe cold leg is approximately Re

.= 1400.For the thermal effects, the thermal diffusivity is com-

puted as κ = 4.05 × 10−3 in LBM units from the valueof viscosity ν because the Prandtl number, a dimension-less number approximating the ratio of momentum viscos-ity and thermal diffusivity, Pr

.= 1.0. The initial fluid do-main temperature is 190.56◦C and the cold water tempera-ture is 31◦C. We use 0.1 in LBM units to represent initialfluid temperature, and the cold water temperature is set as1.63 × 10−2 in the simulation, which keeps the simulationstable, and the effective Rayleigh number (a dimensionlessnumber associated with the heat transfer within the fluid) inthe cold leg is approximately Ra

.= 4.8 × 104.

4.2 Packing of irregular-shaped simulation domain

Figure 5 illustrates the geometry of the PWR, where thefluid flows inside the irregular-shaped vessel. Obviously, the

Fig. 5 The geometry of the PWR. Because of the irregular shape of thereactor vessel, only 5.8% of the cells are useful if we straightforwardlyuse a rectangular region as the simulation domain

empty cells waste much memory space and computationtime in a rectangular simulation domain. In an experiment,we straightforwardly used an 800 × 600 × 480 simulationdomain and found out that only 13 M cells (5.8% of allcells) were nonempty (fluid or boundary) cells. Based onthis observation, we propose an LBM implementation on anirregular-shaped simulation domain with cell classificationand packing. The implementation only stores and computesnonempty cells. Hence, the simulation performance in termsof storage and computation time is significantly improved.

We classify the empty and nonempty cells in a pre-processing stage. First, we mark the cells close to the ves-sel mesh as the boundary cells by the intersection test ofthe mesh and simulation cells. Then, starting from a givenfluid cell (the seed), a region-growing algorithm is imple-mented by a depth-first search to identify all the fluid cells.The nonempty cells are then compactly stored for simula-tion. For the HTLBM, every nonempty cell requires 44 float-ing point numbers to store the simulation data. Thirty ninefloating point numbers store pre-collision particle distrib-utions, post-collision particle distributions, and moments.Two floating point numbers store the temperature of thecurrent and previous simulation steps. Three floating pointnumbers store flow vorticity. Further, in the streaming step(2) of the D3Q13 LBM and the diffusion–advection stepof temperature field (5), the computation of each fluid cellneeds to access data of its 12 neighboring cells. Therefore,we use an indexing table to store the indices of 12 neighbor-ing cells for every fluid cell. The storage used for each fluidcell is 224 bytes. The total storage for the simulation is ap-proximately 2.8 GB. Without the packing method, the totalstorage for the simulation would be 48.1 GB.

4.3 GPU cluster implementation

Even with the cell classification and packing, the compu-tational resource consumption is still extensive for a singleCPU or GPU. We further accelerate our simulation by per-forming the computation on a GPU cluster. The cluster has32 nodes connected by a Gigabit Ethernet. Each node hasan NVIDIA Quadro FX 4500 GPU with 512 MB texturememory, dual Intel Xeon CPUs, 2 GB system memory, anda PCI-Express bus. Our simulation is executed on 12 nodes.As shown in Fig. 6, the simulation space is decomposed into12 rectangular sub-domains (blocks). Each block is assignedto a cluster node. For each block, the data of the nonemptycells inside it, such as density, velocity, particle density func-tions, and temperature, are densely packed into a few 2Dtextures on the GPU.

To simplify the inter-GPU communication, in each blockwe further classify the fluid cells into three categories: inte-rior fluid cells, border fluid cells, and ghost fluid cells. Aninterior fluid cell belongs to the local GPU, and all its neigh-boring cells also belong to the local GPU. A border fluid

Z. Fan et al.

Fig. 6 The simulation space is decomposed into 12 blocks. Each blockis assigned to one cluster node. For each block we only pack the dataof nonempty cells in the GPU memory

Fig. 7 The data structures stored on the GPUs. The blue arrows rep-resent the data transfers during the inter-GPU communication

cell belongs to the local GPU, but some of its neighboringcells belong to remote GPUs. (Inter-GPU communication iscaused by these border cells.) When a cell belongs to a re-mote GPU but some of its neighboring cells belong to thelocal GPU, we store a ghost fluid cell for it on the localGPU. During the simulation, the GPUs compute simultane-ously, and coarse-grained inter-GPU communication occursin the cluster. In every time step, each GPU reads out dataof its border fluid cells and sends the data to the appropri-ate remote GPUs that need the data. After the GPU receivesdata from remote GPUs, it stores the data in the ghost fluidcells. With the ghost fluid cells storing the received data,each GPU only needs to access its local texture memory dur-ing computation. Figure 7 shows the storage of data on the

Table 1 Average computation time (in milliseconds) of a single LBMstep distributed on 12 nodes with and without GPU acceleration

Resolution Non-empty CPU GPU Speedup

cells cluster cluster

800 × 600 × 480 13.4 M 8759 1510 5.8

600 × 450 × 360 5.6 M 3927 729 5.4

400 × 300 × 240 1.7 M 1150 245 4.7

GPUs. The textures on the left pack the simulation data ofall nonempty cells. The region of the textures are dividedinto sub-regions for storing interior fluid cells, border fluidcells, ghost fluid cells, and boundary cells. The blue arrowsin Fig. 7 represent the coarse-grained data transfers duringthe inter-GPU communication. The three RGBA textures onthe right pack the indices of 12 neighboring cells for inte-rior and border fluid cells. This information for ghost cellsis unnecessary because their flow properties are not locallycomputed.

We use the OpenGL and Cg to program the GPUs anduse the Message Passing Interface (MPI) to program thenetwork communication. To initiate the simulation, multi-ple processes are launched in the cluster, each running on acluster node and controlling a local GPU. For network com-munication, the MPI functions MPI_Send and MPI_Recvare used to send or receive data on the network. To read thedata out from the GPU, the corresponding texture is boundto an OpenGL FrameBuffer Object (FBO), and the func-tion glReadPixels is used. Similarly, glWritePixels is usedto write the data to the GPU. The computation sub-steps areimplemented with GPU fragment programs. In operations,the textures of the source data are bound to texture units,while the destination texture to store results becomes ren-derable by being bound to an FBO. Every simulation stepincludes the following sub-steps: LBM collision, inter-GPUcommunication, LBM streaming, boundary condition com-putation, and temperature field evolution. All the computa-tion sub-steps are executed on the GPU in SIMD fashion. Inthe streaming step and the temperature field evolution step,the fragment programs need to access the flow properties ofneighboring cells. The addresses are calculated based on theindices of the neighboring cells stored in the index textures.

Table 1 reports the simulation performance on our clus-ter with and without GPU acceleration. On 12 CPU nodes,the simulation of resolution 800 × 600 × 480 takes 8.8 sec-onds in each simulation step. The GPU cluster implementa-tion using the same number of nodes runs at 1.5 seconds persimulation step. A speedup factor of 5.8 has been achieved.We have also tested two smaller resolution simulations andhave observed speedup factors around 5.

Visual simulation of thermal fluid dynamics in a pressurized water reactor

5 Visualization

5.1 3D volume rendering

The basic tool of our visualization system is a GPU-based3D volume rendering program, which allows the user tointeract with the volumetric temperature field in real-time.We employ a high-quality rendering algorithm, the pre-integrated volume rendering [7]. The transfer function usedin the volume rendering is based on the HSV color model.We store the transfer function in a 1D texture for translatingtemperature to color and opacity. Based on the user intuition,the lowest temperature is assigned a blue hue ( 4

3π ), and thehighest temperature is assigned a red hue (0) in default. Be-cause the user is more interested in the distribution of low-temperature water, the opacity is inversely proportional tothe temperature. The user can change the transfer functionby assigning the colors to temperatures freely in the HSVcolor space. Currently, the HSV color model is used becausethe scientists in nuclear reactor safety analysis are familiarwith it. It is straightforward to change from the HSV to theCIE Lab color model which is more intuitive to human per-ception.

We store the temperature field at each simulation timestep to disks. After the simulation, these data are combinedtogether from multiple cluster nodes, and a single GPU isused to visualize the data. For efficient storage, the float-ing point temperature is converted to unsigned byte datathrough scaling and biasing. To efficiently visualize the datain the irregular-shaped simulation domain, we use multi-ple 3D textures to store multiple blocks of data. The sizesand positions of the blocks have been shown in Fig. 6. Eachblock represents the data copied from one cluster node. Asdescribed in Sect. 4.3, the simulation results are organizedin an unstructured way using indirect addressing. Thus, inpreprocessing of the visualization, we reorganize the sim-ulation results and fill the data into the 3D blocks. Duringthe visualization, the GPU renders the multiple blocks oneby one and composites the rendering results into the finalimage.

Figure 8 shows a sequence of snapshots of the cold waterplume evolution.1 The cold water is injected from the HPIsinto the four cold legs. The cold water then flows down tothe downcomer of the vessel. Due to the interaction betweenthe cold water and the existing hot water, the four plumesappear and affect the temperature of the vessel wall. Finally,the four cold water plumes flow to the bottom plenum of thevessel. They start to merge and induce more turbulent flowsinside the bottom plenum.

The analysts often want to examine the temperature dis-tribution at some specific positions for precise analysis. Ourprogram also provides a slicing option that can show thetemperature distribution on any arbitrary plane with user de-fined position and orientation in the whole modeling space.Figure 9 shows a sequence of snapshots of a particular ver-tical 2D slice cutting in the middle of the vessel.

5.2 Thermal layers rendering

Volume rendering gives a good overview of the entire tem-perature field and shows the interesting turbulent flow. How-ever, due to occlusion and compositing, it does not clearlydepict the internal structure and dynamics of the tempera-ture field, especially the shape and dynamics of the cold wa-ter plumes. Scientists usually have in mind an intuitive un-derstanding of the major structure of the temperature field.We have found that the structure is similar to the conceptof iso-surfaces in the visualization community. Hence weuse iso-surface volume rendering to visualize the structureof the temperature field, called the thermal layers. Our goalis to render multiple translucent iso-surfaces in real-time sothat the user can interact with the temperature field and arbi-trarily add, remove, and change the temperature iso-values.

One method is to explicitly extract the iso-surfaces andthen use a view-independent translucent surface rendering

1All the videos of our visualization results are available at http://www.cs.sunysb.edu/~vislab/projects/amorphous/Plant/. In these videos, wegenerate one frame for every 50 simulation steps.

Fig. 8 A sequence of 3D volume rendering of the simulation results of cold water injected into the reactor vessel. Time t (in seconds) is thesimulation time

Z. Fan et al.

Fig. 9 The simulation results of cold water injected into the reactor vessel on a vertical 2D slice in the middle of the vessel

Fig. 10 The rendering of five thermal layers: (a) an overview, (b) a close-up view showing the thermal layers developed in the cold leg, and(c) a close-up view showing the cold water layer as it penetrates the warm and hot water layers

algorithm, such as depth peeling [9], to display the iso-surfaces. However, the performance of both iso-surface ex-traction and translucent surface rendering greatly dependson the number of iso-values and the complexity of iso-surfaces. For example, currently, the fastest Marching Cubesiso-surface extraction implementation on the GPU [6] canextract a single iso-surface from a volume of 16M voxels ata speed of 4 FPS to 16 FPS (depending on the percentage ofcells intersected by the iso-surface). Our simulation domaincontains 13M nonempty voxels. Supposing that we want toshow ten thermal layers simultaneously, the iso-surface ex-traction will be performance at approximately 0.5 FPS to2 FPS, which does not meet our requirement of real-timespeed. Note that the rendering of such complex multipletranslucent surfaces is also time consuming and will de-crease the frame rate.

We instead employ the iso-surface volume rendering al-gorithm based on the pre-integrated rendering algorithm [7].Our implementation of pre-integrated volume rendering isreused here and only the transfer function is redesignedbased on the selected iso-values. For each iso-value, a “

⋂”

shape is defined in the transfer function. The user can eas-ily add, remove, and change the iso-values. The transfer

Table 2 Performance of our visualization methods tested on aGeForce 8800 GTX. (The image size is 5122. The step size of ray-casting is one grid unit.)

Frames per second

3D volume rendering 17

Slice view rendering 880

Thermal layers rendering 17

Panorama rendering 129

function and the pre-integration table are updated accord-ingly. As shown in the third row of Table 2, our renderingmethod achieves 17 FPS on the GPU. The advantage of thismethod is that the rendering performance is totally inde-pendent of the number and complexity of the thermal lay-ers. Figure 10(a) shows our rendering results of five thermallayers. Compared with the 3D volume rendering in Fig. 8,it shows less turbulence but more clearly depicts the mainstructure of the temperature field and the cold water plumes.Figures 10(b) and 10(c) show two close views of the keylocations, where the temperature field can be checked in de-tail. Figure 10(b) shows the development of thermal layersin the cold leg, where the hot layer is on the top and the cold

Visual simulation of thermal fluid dynamics in a pressurized water reactor

Fig. 11 Panorama rendering (a) with composing and (b) without compositing. In (b), only a slice on the inner wall is rendered. The user can dragthe mouse over the panorama view and see the temperature values of the interesting position in a separate view

layer is at the bottom. Figure 10(c) shows that when the wa-ter flows into the downcomer, the cold water layer penetratesthe warm and hot water layers and touch towards the innerwall of the downcomer (core barrel) as opposed to the reac-tor vessel wall.

5.3 Panorama rendering

Although the 3D views present the entire temperature fieldto the user, some of the information is lost due to occlusionand compositing. In this particular application, an impor-tant part to be analyzed is the small toroidal region betweenthe inner and outer boundary of the downcomer, which isonly a small part of the entire simulation domain. A pos-sible solution is to employ an empty space skipping tech-nique, but it is a complex algorithm and does not solvethe occlusion problem. Instead, we propose a GPU-basedpanorama ray-casting algorithm which parameterizes thetoroidal downcomer and maps it to a rectangle space. Thealgorithm efficiently displays the temperature distribution inthe downcomer in a single image, so that the analyst canview the global distribution without occlusion and can eas-ily find key positions for further investigation. Figure 11(a)demonstrates a panorama volume rendering result image.Figure 11(b) shows the temperature distribution on the sur-face of the vessel downcomer without compositing. Thepanoramic view also results in an intuitive user interface:it allows the analyst to drag the mouse over the view andsee in a separate view the digital values of the temperatureat interesting positions (see Fig. 11(b)).

Figure 12 illustrates the algorithm. The rays are cast fromthe centerline of the vessel, and they are perpendicular tothe centerline. We construct a cylindrical coordinate system(r, θ, y), where θ is the angle between the ray and the x axis.The y axis is the centerline of the vessel. Given a point (u, v)

in parameter space, its corresponding ray segment starts

Fig. 12 Illustration ofpanorama ray-casting. It showsa cutting plane that intersectswith the downcomer. The centerof the circles is a point on thecenterline of the vessel. A ray iscast from the center andaccumulates color and opacityalong the red segment betweenthe inner circle and outer circle

from (r0,2πu,v) and ends at (r1,2πu,v), where r0 and r1

are the radii of inner and outer cylinders, respectively. Inpractice, a rectangle is drawn to the screen with proper tex-ture coordinates ([0..1]×[0..1]). The fragment program firstcalculates the ray direction and the origin for each fragment.Then, a single pass ray casting algorithm [14] is used to cal-culate the accumulated color and transparency on each ray.The radii r0 and r1 are used for empty space skipping andray termination. As shown in Table 2, the performance ofthe panorama rendering is approximately an order of mag-nitude higher than the 3D rendering methods (volume ren-dering and thermal layers rendering) because only a smallportion of voxels is processed.

5.4 Statistical analysis

For a complete PWR safety analysis, our thermal fluid dy-namics simulation results can be converted to the inputs ofa structural simulation for further examination of the vesselperformance. In our current system, we provide statisticalanalysis tools to help the analysts identify the key regionsbefore the structural simulation. Intuitively, the regions ofthe lowest temperature and/or highest temperature gradientare of most interest. Hence, we record the minimum temper-ature and maximum gradient magnitude for each lattice cellduring the simulation. The volumes of minimum tempera-ture and maximum temperature gradient magnitude are then

Z. Fan et al.

Fig. 13 In (a) and (b), the colors represent the minimum temperature over the simulation time, with volume rendering and surface rendering,respectively. In (c) and (d), the colors represent the maximum temperature gradient over the simulation time

Fig. 14 (a) Four points (shown in the blue rectangle) in the cold leg defined by the user. (b) The plots of the temperature history at these points

downloaded to a GPU for visualization. The GPU rendersthe PWR geometry with the color map based on the datasampled from the 3D textures that store the minimum tem-perature and maximum temperature gradient. Figures 13(a)and 13(b) show the volume rendering and polygonal render-ing images of the minimum temperature over the simulationtime. Figures 13(c) and 13(d) show the maximum tempera-ture gradient magnitude.

For positions of special interest to the analyst, the systemcan output accurate temperature readings for further analy-sis. Figure 14(a) shows four points specified by the user in-side the cold leg. For these points, the temperature history isplotted in Fig. 14(b). The figure shows that in the cold leg theupper points have higher temperature than the lower points.This phenomenon agrees with our thermal layers renderingresult shown in Fig. 10(b).

6 Conclusions

We pioneered the successful use of the lattice-Boltzmannmethod in analyzing the cold water plumes in the reactor

vessel downcomer of a power plant. The LBM is powerfuland flexible for the analysis due to its great ability for han-dling complex 3D geometries as boundary conditions andcoupling temperature evolution. Furthermore, it can be eas-ily adapted for GPU and GPU cluster acceleration becauseof its local and linear operations. Therefore, it provides apreferred framework to model a PTS-related case with var-ious boundary conditions, thermal profiles, and model con-figurations. With a GPU cluster, we manage to accommo-date the huge irregular-shaped geometry into the GPU mem-ories to accelerate the simulation. In addition, we have ex-perimented with a set of interactive GPU-based methods forvisualizing the cold water plumes. Pre-integrated volumerendering has been used to render the 3D views of the entiresimulation domain. Thermal layers rendering clearly revealsthe structures and dynamics of the temperature field and coldwater plumes. Panorama rendering provides a fast and effi-cient way to analyze the water body inside the downcomerwithout occlusion.

In the future, we will port our simulation to a newer GPUcluster using the NVIDIA Compute Unified Device Archi-tecture (CUDA) programming toolkit. We plan to connect

Visual simulation of thermal fluid dynamics in a pressurized water reactor

our simulation with a structural simulation for further ves-sel performance analysis. Further experiments and valida-tion tests will be conducted for the system.

Acknowledgements The work has been supported by NSFCCF0702699 and funding from the U.S. Nuclear Regulatory Com-mission (NRC) through Information Systems Laboratories, Inc (ISL).Figures 1 and 2 were adapted from [20]. We would also like to thankAutodesk, Inc. for donating Maya 2008 to the Center for Visual Com-puting. Maya 2008 has been used to create the PWR geometry for thesimulation in this research project.

References

1. Bolz, J., Farmer, I., Grinspun, E., Schröder, P.: Sparse matrixsolvers on the GPU: Conjugate gradients and multigrid. ACMTrans. Graph. 22(3), 917–924 (2003)

2. Chen, S., Doolen, G.: Lattice Boltzmann method for fluid flows.Annu. Rev. Fluid Mech. 30, 329–364 (1998)

3. Chu, N.S.-H., Tai, C.: Moxi: Real-time ink dispersion in absorbentpaper. ACM Trans. Graph. 24(3), 504–511 (2005)

4. Crane, K., Llamas, I., Tariq, S.: Real-time simulation and render-ing of 3D fluids. In: Nguyen, H. (ed.) GPU Gems 3, pp. 633–676.Addison–Wesley, Reading (2007)

5. D’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand, P.,Luo, L.-S.: Multiple-relaxation-time lattice Boltzmann models inthree dimensions. R. Soc. Lond. Philos. Trans., Ser. A 360(1792),437–451 (2002)

6. Dyken, C., Ziegler, G., Theobalt, C., Seidel, H.-P.: Histopyramidsin iso-surface extraction. Technical Report, Max Planck Inst. fürInfor. (2007)

7. Engel, K., Kraus, M., Ertl, T.: High-quality pre-integrated volumerendering using hardware-accelerated pixel shading. ACM SIG-GRAPH/EUROGRAPHICS Workshop on Graphics Hardware,pp. 9–16 (2001)

8. EricksonKirk, M., Junge, M., Arcieri, W., Bass, B., Beaton, R.,Bessette, D., Chang, T., Dickson, T., Fletcher, C., Ko-laczkowski, A., Malik, S., Mintz, T., Pugh, C., Simonen, F.,Siu, N., Whitehead, D., Williams, P., Woods, R., Yin, S.: Tech-nical basis for revision of the pressurized thermal shock (PTS)screening limit in the PTS rules (10 CFR 50.61). NUREG-1806, 1(2006)

9. Everitt, C.: Interactive order-independent transparency. TechnicalReport, NVIDIA Corporation (2001)

10. Fan, Z., Qiu, F., Kaufman, A., Yoakum-Stover, S.: GPU clusterfor high performance computing. In: ACM/IEEE SupercomputingConference, p. 47 (2004)

11. Fan, Z., Qiu, F., Kaufman, A.: Zippy: A framework for computa-tion and visualization on a GPU cluster. Comput. Graph. Forum27(2), 341–350 (2008)

12. Goodnight, N., Woolley, C., Lewin, G., Luebke, D.,Humphreys, G.: A multigrid solver for boundary valueproblems using programmable graphics hardware. In: ACMSIGGRAPH/Eurographics Workshop on Graphics Hardware,pp. 102–111 (2003)

13. Harris, M., Baxter, W.V., Scheuermann, T., Lastra, A.: Simula-tion of cloud dynamics on graphics hardware. In: ACM SIG-GRAPH/Eurographics Workshop on Graphics Hardware, pp. 92–101 (2003)

14. Krüger, J., Westermann, R.: Acceleration techniques for GPU-based volume rendering. In: IEEE Vis., pp. 287–292 (2003)

15. Krüger, J., Westermann, R.: Linear algebra operators for GPU im-plementation of numerical algorithms. ACM Trans. Graph. 22(3),908–916 (2003)

16. Lallemand, P., Luo, L.-S.: Theory of the lattice Boltzmannmethod: Acoustic and thermal properties in two and three dimen-sions. Phys. Rev. E 68(3), 036706 (2003)

17. Li, F., Modarres, M.: Probabilistic modeling for fracture mechanicstudies of reactor vessels with characterization of uncertainties.Nucl. Eng. Des. 235(1), 1–19 (2005)

18. Li, W., Wei, X., Kaufman, A.: Implementing lattice Boltzmanncomputation on graphics hardware. Vis. Comput. 19(7–8), 444–456 (2003)

19. Li, W., Fan, Z., Wei, X., Kaufman, A.: Flow simulation with com-plex boundaries. In: Pharr, M. (ed.) GPU Gems 2: ProgrammingTechniques for High-Performance Graphics and General-PurposeComputation, pp. 747–764. Addison–Wesley, Reading (2005)

20. Lobner, P., Donahoe, C., Vavallin, C.: Overview and compari-son of U.S. commercial nuclear power plants. Technical ReportNUREG/CR-5640 (1990)

21. Martin, A., Bellet, S.: CFD-tools qualification for thermal-hydraulics pressurized thermal shock analysis. J. Press. VesselTechnol. 125(4), 418–424 (2003)

22. Owens, J.D., Luebke, D., Govindaraju, N., Harris, M., Krüger, J.,Lefohn, A.E., Purcell, T.J.: A survey of general-purpose computa-tion on graphics hardware. Comput. Graph. Forum 26(1), 80–113(2007)

23. Succi, S.: The Lattice Boltzmann Equation for Fluid Dynamicsand Beyond. Numerical Mathematics and Scientific Computation.Oxford University Press, London (2001)

24. Thürey, N., Ruede, U.: Free surface lattice-Boltzmann fluid simu-lations with and without level sets. Vis. Model. Vis., pp. 199–207(2004)

25. van Treeck, C., Rank, E., Krafczyk, M., Tölke, J., Nachtwey, B.:Extension of a hybrid thermal LBE scheme for large-eddy sim-ulations of turbulent convective flows. Comput. Fluids 35(8–9),863–871 (2006)

26. Wei, X., Li, W., Mueller, K., Kaufman, A.: Simulating fire withtexture splats. IEEE Vis., pp. 227–234 (2002)

27. Wei, X., Zhao, Y., Fan, Z., Li, W., Qiu, F., Yoakum-Stover, S.,Kaufman, A.: Lattice-based flow field modeling. IEEE Trans. Vis.Comput. Grap. 10(6), 719–729 (2004)

28. Zhao, Y.: Lattice Boltzmann based PDE solver on the GPU. Vis.Comput. 24(5), 323–333 (2008)

29. Zhao, Y., Han, Y., Fan, Z., Qiu, F., Kuo, Y., Kaufman, A.E.,Mueller, K.: Visual simulation of heat shimmering and mirage.IEEE Trans. Vis. Comput. Graph. 13(1), 179–189 (2007)

30. Zhu, H., Liu, X., Liu, Y., Wu, E.: Simulation of miscible binarymixtures based on lattice Boltzmann method. Comput. Animat.Virtual Worlds 17(3–4), 403–410 (2006)

Zhe Fan received the BS degreein computer science from the Uni-versity of Sciences and Technol-ogy of China in 1998, the MS de-gree in computer science from Chi-nese Academy of Sciences in 2001,the PhD degree in computer sci-ence from Stony Brook Universityin 2008. His research is focused onflow simulation and visualization onGPUs and GPU clusters. He is nowwith Google Inc.

Z. Fan et al.

Yu-Chuan Kuo received the BS de-gree in computer science and in-formation engineering from the Na-tional Taiwan University in 2002.He received MS degrees in the De-partment of Computer Science andthe Department of Applied Mathe-matics and Statistics at Stony BrookUniversity in 2007.

Ye Zhao is currently an assistantprofessor in the Department of Com-puter Science at the Kent State Uni-versity. He received his BE and MSdegrees in computer science fromthe Tsinghua University of China in1997 and 2000. He further receivedhis PhD degree in computer sci-ence from the Stony Brook Univer-sity in 2006. His research interestsinclude physically based modelingand visualization, GPGPU, imageand geometric processing, and vol-ume graphics. He is a member ofthe IEEE. For more information, seehttp://www.cs.kent.edu/~zhao.

Feng Qiu received the BS degree incomputer science from the PekingUniversity of China in 1997, theME degree in computer engineeringfrom the Chinese Academy of Sci-ences in 2000, and the PhD degreein computer science from StonyBrook University in 2008. His re-search is focused on hardware ac-celerated rendering. He is now a re-search scientist in Siemens Corpo-rate Research.

Arie Kaufman received the BSdegree (1969) in mathematics andphysics from the Hebrew Universityof Jerusalem, Israel, the MS degree(1973) in computer science from theWeizmann Institute of Science, Re-hovot, Israel, and the PhD degree(1977) in computer science from theBen-Gurion University, Israel. He isa distinguished professor and chairof the Computer Science Depart-ment, chief scientist of the Centerof Excellence in Wireless and Infor-mation Technology (CEWIT), andthe director of the Center for Visual

Computing (CVC) at the State University of New York at Stony Brook(SBU). He is an IEEE fellow, a member of the IEEE Computer So-ciety, and the recipient of IEEE Visualization Career Award (2005).He further received the IEEE Outstanding Contribution Award (1995),the ACM Service Award (1998), the IEEE CS Meritorious ServiceAward (1999), was a member of the European Academy of Sciences(2002), the State of New York Entrepreneur Award (2002), the IEEEHarold Wheeler Award (2004), and the State of New York Innova-tive Research Award (2005). Dr. Kaufman was the founding Editor-in-Chief of IEEE Transactions on Visualization and Computer Graph-ics (TVCG), 1995–1998. He has been the cofounder, papers/programcochair, and member of the steering committee of IEEE Visualiza-tion Conferences; cofounder/chair of Volume Graphics Workshops;cochair for Proceedings of Eurographics/SIGGRAPH Graphics Hard-ware Workshops, the papers/program cochair for Proceedings of theACM Volume Visualization Symposia. He previously chaired and iscurrently a director of the IEEE CS Technical Committee on Vi-sualization and Graphics. He has conducted research and consultedfor more than 35 years specializing in volume visualization, graph-ics architectures, algorithms, and languages, virtual reality, user inter-faces, multimedia, and their applications. For more information, seehttp://www.cs.sunysb.edu/~ari.

William Arcieri holds a M.S. inNuclear Engineering from ColumbiaUniversity and a M.S. in ComputerScience from Johns Hopkins Uni-versity. He is employed by ISL, Inc.,where he has been involved with nu-clear power safety work for the past15 years.