CEE 618 Scientiﬁc Parallel Computing (Lecture 12)...DPD (Dissipative Particle Dynamics) = Simulation method for Brownian motion of multiple particles using (approximate)pair-wise

CEE 618 Scientific Parallel Computing (Lecture 12)Dissipative Hydrodynamics (DHD)

Albert S. Kim

Department of Civil and Environmental EngineeringUniversity of Hawai‘i at Manoa

2540 Dole Street, Holmes 383, Honolulu, Hawaii 96822

1 / 26

Particle Dynamics

Outline

1 Particle DynamicsIntroductionBrownian DynamicsStokesian DynamicsLab work and Project

2 Raster3DVisualizing Spheres

2 / 26

Particle Dynamics Introduction

What is Particle Dynamics?

A study of motion of multiple particles,influenced by forces and torques

3 / 26


What is the force?

FORCEA push or pull that can cause an object with mass to accelerateNewton’s second law:

F = ma

Acceleration:

a =dv

dt=d2r

dt2

ENERGYA scalar physical quantity that is a property of objects andsystems which is conserved by natureThe ability to do work:

E = −∫ r2r1

F · dr

only if F = F(r).4 / 26


Statistical Mechanical Approaches

1 Nano-scale (10−9 m)MD (Molecular Dynamics) = Deterministic simulation of solvingNewton’s second law for ion species

2 Nano to Micro-scale (10−6 m)BD (Brownian Dynamics) = Updated simulation protocol of MD forions in a fluid medium, but more applied to volumeless (point)colloidal/nano-particles: Random Forces/TorquesDPD (Dissipative Particle Dynamics) = Simulation method forBrownian motion of multiple particles using (approximate) pair-wisehydrodynamics.

3 Nano to Meso-scale (10−3 m)SD (Stokesian Dynamics) = Accurate simulation method formicro-hydrodynamics of spherical particlesDHD = General simulation method for micro-hydrodynamics ofBrownian and non-Brownian particles

5 / 26

Particle Dynamics Brownian Dynamics

Brownian Dynamics: Langevin’s Equation

The Langevin equations for the system of N Brownian particles:for particle i interacting with j’s

ṗi = miv̇i = Fi (r) +∑j

(−) ξijvj +∑j

αijfj

1 Molecular Dynamics for conservative forces/torques2 Stokesian Dynamics for hydrodynamic forces/torques3 Dissipative Particle Dynamics for stochastic forces/torques* On the average hydrodynamic ≈ stochastic

pi = mivi is the momentum,ξij is the hydrodynamic friction tensor,Fi is the sum of inter-particle and external forces, and∑

j αijfj represents the randomly fluctuating force exerted on aparticle by the surrounding fluid: negligible if particles are muchbigger than 1.0 µm.

6 / 26


Properties of Random Fluctuating Force, fi

1 Time average is zero:〈fi〉 = 0 (1)

2 Independently exerted on i and j particle of different positions(i.e., ri and rj) and at different times (i.e., t and t′)

〈fi (t) fj(t′)〉 = 2δijδ

(t− t′

)(2)

3 δ is the Dirac-delta function:δij = 0 if i 6= j; and δij = 1 if i = j;δ (t− t′) = 0 if t 6= t′; and δ (t− t′) = 1 if t = t′.

4 Related to the friction coefficient

ξij =1

kBT

∑k

αikαjk (3)

indicating α ∼√ξ.

7 / 26


Brownian DynamicsIntegration of the Langevin equation gives the time evolution equation:

ri (t+ ∆t) = ri (t) +∑j

Dij (t)

kBT· Fj ∆t+ (∇ ·D) ∆t+ ∆rGi (4)

where the components of ∆rGi are random displacements selectedfrom 3N variate Gaussian distribution with zero means andcovariance matrix

〈∆rGi 〉 = 0 and 〈∆rGi ∆rGj 〉 = 2Dij∆t (5)The Oseen tensor (crude approximation) is given by

Dij =kBT

6πηa1, for i = j (6a)

=kBT

8πηrij

(1 +

rijrijr2ij

), for i 6= j (6b)

and one calculates ∇ ·D = 0. If Fj ≈ 0, the random motion isdominant in multi-particle dynamics: ∆rGi ∝

√∆t.

8 / 26


Brownian Dynamics (BD)

Langevin equation1 with inter-particle (conservative) forces fP ,drag forces fH = −ξv, and random Brownian forces fB

mdv

dt= fP + fH + fB (t) (7a)

fH = −ξv (7b)〈fB(t)〉 = 0 (7c)

〈fB(0) · fB(t)〉 = 6ξkBTδ (t) (7d)

1Ermak and McCammon, J. Chem. Phys. 69 (1978) 1352-1360; Langevin,C. R. Acad. Sci. (Paris) 146 (1908) 530-533

9 / 26


e.g., a falling body in liquid with x(0) = 0 & v(0) = 0ma = −mg − βv + fB (t)

10 / 26

Particle Dynamics Stokesian Dynamics

Stokesian Dynamics: Langevin’s Equation

The Langevin equations for the system of N force-free, non-Brownianparticles

ṗi = miv̇i = −∑j

ξij (vj − U) ≡ FH

FH is the hydrodynamic forces/torques,pi = mivi is the momentum, andξij is the hydrodynamic friction tensor.

If particles are at rest,

U = M∞ · FH (8)FH = R∞ ·U (9)R∞ = (M∞)−1 (10)

where U is the translational/rotational velocity vector, and M∞ andR∞ are the grand mobility and grand resistance matrixes, respectively.R∞ is dependent on particle positions and calculated as an inversematrix of M∞.

11 / 26


Stokesian Dynamics (SD)

Particles translate and rotate in a fluid field of

V = U∞ + r ×Ω∞ + E∞ : r

where U∞ is the uni-directional flow; and the vorticity Ω∞ and rate ofstrain E∞ are represented as

Ω∞ = 12∇× V (r)

E∞ij =1

2

(∂Vi∂xj

+∂Vj∂xi

)= 12 (∂jVi + ∂iVj) = Eji

respectively. If no shear, E∞ = 0

12 / 26


Hydrodynamic Force Calculation with upflow U

U = M∞ · FH and FH6Np×1 =?where U6Np×1 is the relative velocities, FH6Np×1 is the hydrodynamic forces onparticles, and M∞6Np×6Np is the grand mobility matrix.

Hydrodynamic Force Visualization: Two Examples

6443 ==pN 303,11=pN

Hassonjee, Q., Ganatos, P., and Pfeffer, R., J. Fluid Mech., 197, 1-37 (1988)

70 minutes of running timeusing 25 processors

13 / 26


Parallel Computation of SD Simulation

6Np × 6Np

Parallel computing of large matrixes using 1,600 processors (outof 5400) of Jaws at Maui High Performance Computing Center(MHPCC)

1 The grand mobility matrix M∞ has a dimension 6Np × 6Np2 Np = 40

3 = 64, 000 −→ (6Np)2 = 147 billion elements3 Memory = 1.18 TB −→ 738 MB per processor4 Time to calculate FH = 47 min.

Using Tachyon at KISTI1 Np = 384

2 = 147, 456 −→ (6Np)2 = 783 billion elements2 4096 cores, 6.3 TB, and 16.5 hours

14 / 26


SD Simulation

When particles are at rest, and a uniform upflow approaches Npparticles with a constant velocity U0 = 1 (in dimensionless unit),

U0 = M∞ · FH

then the hydrodynamic forces acting on the particles arecalculated as F6Np×1.Each particle has six component of in F6Np×1.

1 F1 − F3 are forces on particle 1 in x, y, and z-directions, andF4 − F6 are torques on particle 1 in x, y, and z-directions.

2 F7 − F9 are forces on particle 2 in x, y, and z-directions, andF10 − F12 are torques on particle 2 in x, y, and z-directions.

3 And so forth ...For upward velocity, Uj = 1 if j = 3 + 6(i− 1) otherwise Uj = 0:non-zero Uj for j = 3, 9, 15, · · · .An example calculation was included in Hassonjee, Q., Ganatos, P.,& Pfeffer, R. (1988). J. Fluid Mech., 197, 1–37.

15 / 26


Cubic configuration of 64 particles with D/a = 16.12

1 0.000000 0.000000 0.000000 2 16.120000 0.000000 0.000000 3 32.240000 0.000000 0.000000 4 48.360000 0.000000 0.000000 5 0.000000 16.120000 0.000000 6 16.120000 16.120000 0.000000 7 32.240000 16.120000 0.000000 8 48.360000 16.120000 0.000000 9 0.000000 32.240000 0.00000010 16.120000 32.240000 0.00000011 32.240000 32.240000 0.00000012 48.360000 32.240000 0.00000013 0.000000 48.360000 0.00000014 16.120000 48.360000 0.00000015 32.240000 48.360000 0.00000016 48.360000 48.360000 0.000000

Figure: 4× 4× 4 array

16 / 26


Force calculation: D/a = 16.12: degenerated z-forces

17 / 26


Results: Fx, Fy, Fz, Tx, Ty, Tz for 64 particles

1 Which column is always positive and why?2 Compare the fourth column-values with Fz ’s in the previous page.3 How to get unique values of Fz in the fourth column?

Use cat, cut, and sort. 1 -0.57117809E-01 -0.57117809E-01 0.47000409E+00 -0.64404449E-02 0.64404449E-02 0.00000000E+00 2 -0.16968012E-01 -0.68096668E-01 0.40061967E+00 -0.74043259E-02 0.14526390E-02 -0.65597849E-04 3 0.16968012E-01 -0.68096668E-01 0.40061967E+00 -0.74043259E-02 -0.14526390E-02 0.65597849E-04 4 0.57117809E-01 -0.57117809E-01 0.47000409E+00 -0.64404449E-02 -0.64404449E-02 -0.27105054E-18 5 -0.68096668E-01 -0.16968012E-01 0.40061967E+00 -0.14526390E-02 0.74043259E-02 0.65597849E-04 6 -0.19843350E-01 -0.19843350E-01 0.31979468E+00 -0.16786973E-02 0.16786973E-02 0.13552527E-19 7 0.19843350E-01 -0.19843350E-01 0.31979468E+00 -0.16786973E-02 -0.16786973E-02 -0.27105054E-19 8 0.68096668E-01 -0.16968012E-01 0.40061967E+00 -0.14526390E-02 -0.74043259E-02 -0.65597849E-04 9 -0.68096668E-01 0.16968012E-01 0.40061967E+00 0.14526390E-02 0.74043259E-02 -0.65597849E-04 10 -0.19843350E-01 0.19843350E-01 0.31979468E+00 0.16786973E-02 0.16786973E-02 -0.18973538E-18 11 0.19843350E-01 0.19843350E-01 0.31979468E+00 0.16786973E-02 -0.16786973E-02 -0.27105054E-19 12 0.68096668E-01 0.16968012E-01 0.40061967E+00 0.14526390E-02 -0.74043259E-02 0.65597849E-04 13 -0.57117809E-01 0.57117809E-01 0.47000409E+00 0.64404449E-02 0.64404449E-02 0.54210109E-19 14 -0.16968012E-01 0.68096668E-01 0.40061967E+00 0.74043259E-02 0.14526390E-02 0.65597849E-04 15 0.16968012E-01 0.68096668E-01 0.40061967E+00 0.74043259E-02 -0.14526390E-02 -0.65597849E-04 16 0.57117809E-01 0.57117809E-01 0.47000409E+00 0.64404449E-02 -0.64404449E-02 0.54210109E-19 17 -0.18100733E-01 -0.18100733E-01 0.41733240E+00 -0.69054373E-02 0.69054373E-02 -0.54210109E-19 18 -0.58771938E-02 -0.21903566E-01 0.34414116E+00 -0.78628707E-02 0.14394909E-02 -0.30656360E-04 19 0.58771938E-02 -0.21903566E-01 0.34414116E+00 -0.78628707E-02 -0.14394909E-02 0.30656360E-04 20 0.18100733E-01 -0.18100733E-01 0.41733240E+00 -0.69054373E-02 -0.69054373E-02 -0.28460307E-18 21 -0.21903566E-01 -0.58771938E-02 0.34414116E+00 -0.14394909E-02 0.78628707E-02 0.30656360E-04 22 -0.69521930E-02 -0.69521930E-02 0.26186929E+00 -0.16218298E-02 0.16218298E-02 -0.54210109E-19 23 0.69521930E-02 -0.69521930E-02 0.26186929E+00 -0.16218298E-02 -0.16218298E-02 0.60986372E-19 24 0.21903566E-01 -0.58771938E-02 0.34414116E+00 -0.14394909E-02 -0.78628707E-02 -0.30656360E-04 25 -0.21903566E-01 0.58771938E-02 0.34414116E+00 0.14394909E-02 0.78628707E-02 -0.30656360E-04 26 -0.69521930E-02 0.69521930E-02 0.26186929E+00 0.16218298E-02 0.16218298E-02 0.23716923E-19 27 0.69521930E-02 0.69521930E-02 0.26186929E+00 0.16218298E-02 -0.16218298E-02 0.64374504E-19 28 0.21903566E-01 0.58771938E-02 0.34414116E+00 0.14394909E-02 -0.78628707E-02 0.30656360E-04 29 -0.18100733E-01 0.18100733E-01 0.41733240E+00 0.69054373E-02 0.69054373E-02 0.94867690E-19 30 -0.58771938E-02 0.21903566E-01 0.34414116E+00 0.78628707E-02 0.14394909E-02 0.30656360E-04 31 0.58771938E-02 0.21903566E-01 0.34414116E+00 0.78628707E-02 -0.14394909E-02 -0.30656360E-04 32 0.18100733E-01 0.18100733E-01 0.41733240E+00 0.69054373E-02 -0.69054373E-02 0.10842022E-18 33 0.18100733E-01 0.18100733E-01 0.41733240E+00 -0.69054373E-02 0.69054373E-02 0.13552527E-19 34 0.58771938E-02 0.21903566E-01 0.34414116E+00 -0.78628707E-02 0.14394909E-02 0.30656360E-04 35 -0.58771938E-02 0.21903566E-01 0.34414116E+00 -0.78628707E-02 -0.14394909E-02 -0.30656360E-04 36 -0.18100733E-01 0.18100733E-01 0.41733240E+00 -0.69054373E-02 -0.69054373E-02 0.12197274E-18 37 0.21903566E-01 0.58771938E-02 0.34414116E+00 -0.14394909E-02 0.78628707E-02 -0.30656360E-04 38 0.69521930E-02 0.69521930E-02 0.26186929E+00 -0.16218298E-02 0.16218298E-02 0.64374504E-19 39 -0.69521930E-02 0.69521930E-02 0.26186929E+00 -0.16218298E-02 -0.16218298E-02 0.64374504E-19 40 -0.21903566E-01 0.58771938E-02 0.34414116E+00 -0.14394909E-02 -0.78628707E-02 0.30656360E-04

18 / 26


Directions of force/torque: Fx, Fy, Fz, Tx, Ty, Tz

Exerted on each particle with upflow, U = +1 (↑). 1 -0.57117809E-01 -0.57117809E-01 0.47000409E+00 -0.64404449E-02 0.64404449E-02 0.00000000E+00 2 -0.16968012E-01 -0.68096668E-01 0.40061967E+00 -0.74043259E-02 0.14526390E-02 -0.65597849E-04 3 0.16968012E-01 -0.68096668E-01 0.40061967E+00 -0.74043259E-02 -0.14526390E-02 0.65597849E-04 4 0.57117809E-01 -0.57117809E-01 0.47000409E+00 -0.64404449E-02 -0.64404449E-02 -0.27105054E-18 5 -0.68096668E-01 -0.16968012E-01 0.40061967E+00 -0.14526390E-02 0.74043259E-02 0.65597849E-04 6 -0.19843350E-01 -0.19843350E-01 0.31979468E+00 -0.16786973E-02 0.16786973E-02 0.13552527E-19 7 0.19843350E-01 -0.19843350E-01 0.31979468E+00 -0.16786973E-02 -0.16786973E-02 -0.27105054E-19 8 0.68096668E-01 -0.16968012E-01 0.40061967E+00 -0.14526390E-02 -0.74043259E-02 -0.65597849E-04

19 / 26

Particle Dynamics Lab work and Project

Lab work

SD code code for hydrodynamic force/torque calculation is in/opt/cee618s13/class12/hasonjee/

20 / 26

Raster3D

Outline

1 Particle DynamicsIntroductionBrownian DynamicsStokesian DynamicsLab work and Project

2 Raster3DVisualizing Spheres

21 / 26

Raster3D Visualizing Spheres

Raster3D

http://skuld.bmsc.washington.edu/raster3d/

1 Raster3D is a set of tools for generating high quality raster imagesof proteins or other molecules.

2 The core program renders spheres, triangles, cylinders, andquadric surfaces with specular highlighting, Phong shading, andshadowing.

22 / 26

http://skuld.bmsc.washington.edu/raster3d/


Example 1

1 Copy all the files from/opt/cee618s13/class12/raster3d/example1/to your own directory.

2 Type and enter: qsubtraster_ex1.pbs3 This pbs script will execute example1h.script and generate an

image file, example1h.tff

23 / 26


Sphere configuration: 6× 6× 6 array

Under ‘/mnt/home/albertsk/UHTraining/cee618-sp2012/class09/DHD’1 In “sHsnj_obsd_fts_64.f”

To rotate image change Euler angles of alpha0, beta0, andgamma0.To change the distance between the center and your eyes, controldistance “sHsnj_obsd_fts_64.f”.

2 “Raster3Dspheres.f” is included in the main code“sHsnj_obsd_fts_64.f”.

3 There will be three output files from this serial run:1 “sForceFTS.dat” stores force/torque calculation data.2 “sCoordXYZ.dat” includes (x, y, z) coordinates of Np particles.3 “sCoordXYZ.r3d” contains Raster3D format coordinate data,

translated to the center of mass.

24 / 26


How to generate an image

1 Copy all the files in /opt/cee618s13/class12/dhd-raster3d/ to yourown directory.

2 Execute$ make$ maketrun

3 Then, a file like “sCoordXYZ.tff” will be generated.4 Download the .tff file and view it.

25 / 26


Raster file: sCoordXYZ.r3d, x, y, z, a, and 3 more

1 Example of material properties and file indirection2 80 64 tiles in x,y3 8 8 pixels (x,y) per tile4 4 3x3 virtual pixels -> 2x2 pixels5 0 0.1 0 background colour6 T cast shadows7 25 Phong power8 0.15 secondary light contribution9 0.05 ambient light contribution

10 0.25 specular reflection component11 4.0 eye position12 1 1 1 main light source position13 0.578E+00 -0.259E+00 0.483E+00 0.000E+0014 0.224E+00 0.966E+00 0.129E+00 0.000E+0015 -0.500E+00 0.000E+00 0.866E+00 0.000E+0016 0.000E+00 0.000E+00 0.000E+00 0.900E+0217 3 mixed objects18 *19 *20 *21 # Draw a bunch of spheres22 #23 #24 #25 @orange.r3d26 227 -.241800E+02 -.241800E+02 -.241800E+02 0.100000E+01 0.100000E+01 0.100000E+01 0.100000E+0128 @green.r3d29 230 -.806000E+01 -.241800E+02 -.241800E+02 0.100000E+01 0.100000E+01 0.100000E+01 0.100000E+0131 @blue.r3d32 233 0.806000E+01 -.241800E+02 -.241800E+02 0.100000E+01 0.100000E+01 0.100000E+01 0.100000E+0134 @red.r3d

26 / 26

Particle DynamicsIntroductionBrownian DynamicsStokesian DynamicsLab work and Project

Raster3DVisualizing Spheres

Documents

CEE 618 Scientiﬁc Parallel Computing (Lecture 12)...DPD (Dissipative Particle Dynamics) = Simulation method for Brownian motion of multiple particles using (approximate)pair-wise