Simulation. UESTC Chengdu (China) - unex.estsc.unex.es/~tabo/taboada_chengdu_2011.pdf · 2011 International Workshop on Electromagnetic Theory, Modeling, and Simulation UESTC Chengdu

Universidad de Extremadura

2011 International Workshop on Electromagnetic Theory, Modeling, and Simulation UESTC Chengdu (China)

Supercomputer solution of extremely large problems in electromagnetics

J.M. Taboada

University of Extremadura, Cáceres, Spain

[email protected]

2011 International Workshop on Electromagnetic Theory, Modeling, and Simulation. UESTC Chengdu (China)

mailto:[email protected]�mailto:[email protected]�



Research team

University of Extremadura

José Manuel Taboada VarelaLuis Landesa PorrasJavier Rivero Campos

University of Vigo

Fernando Obelleiro BasteiroJosé Luis Rodríguez RodríguezMarta Gómez Araújo

Colaboration with

Supercomputing Center of Galicia (CESGA)Supercomputing Center of Extremadura (CénitS)

Spanish Government Projects:TEC2008-06714-C02-02CONSOLIDER-INGENIO2010 CSD2008-00068ICTS-2009-40 Junta de Extremadura (project GR10126).



Outline

Parallel FMM and MLFMAParallel FMM-FFTParallel MLFMA-FFTSurface integral-equation (SIE) formulation for generalized media: left-handed metamaterials (LHM) and plasmonic media



Method of Moments.Computational cost

Real life problems imply the solution of systems with millions of unknowns

Solving with matrix factorization

O(N2) in memoryO(N3) in CPU time

⋅ =Z I V

Solving with iterative methods

O(N2) in memoryO(N2) in CPU time

⋅ =Z I V

Required number of unknowns N

N grows proportional to the square of the frequency

S. M. Rao, D. R.Wilton, and A. W. Glisson, “Electromagnetic scattering by surfaces of arbitrary shape,” IEEE Trans. Antennas Propag. 30(3), 409-418 (1982).



Method of Moments

Method of Moments.RCS of an Airbus A-380 at 1.2 GHz

Memory > 25 PBCPU time

Setup: several decadesSolution with factorization: several thousands of yearsSolution with iterative solver: several decades

Setup and solution in a few hours using 128 parallel processors.

Fast Multipole Methods and parallel computers



Fast Multipole Method (FMM)

O(N1.5)O(N2)

O(N1.5)

Computational cost

O(N1.5)



Multilevel Fast Multipole Algorithm(MLFMA)

Computational cost

O(N log N)

J. M. Song, C. C. Lu, and W. C. Chew, “Multilevel fast multipole algorithm for electromagnetic scattering by large complex objects,” IEEE Transactions on Antennas and Propagation 45, pp. 1488-1493 (1997).



Parallelization of codes for mixed-memory HPC supercomputers

Finis Terrae (CESGA)142 cc-NUMA Integrity rx7640 nodes16 processor cores and 128 GB each.INFINIBAND at 20Gbps

LUSITANIA (CénitS)2 Superdome Integrity nodes128 processor cores and 1,024GB each.



Importance of scalability

Scalability is the ability of a code to take benefit from the use of large parallel computers and supercomputers.

Important issues to get a high-scalability code

Workload balancing among parallel processors Data localityMemory footprintInter-node communication

Besides a proper parallelization strategy, the natural scaling properties of the selected algorithm are of great importance.



FMM parallelization

Parallelization by groups

Load unbalanceHigh requirements of inter-node communications.

Parallelization in k-space

k-space samples are independent one of anotherGood data localityLow need of internode communicationPerfect load balance



MLFMA parallelization

k-space samples are not independent due to interpolation and anterpolation.

Parallel scalability drawbacks

Load imbalanceHigh memory footprintHigher inter-node communication requirements



Pros and cons of parallel FMM and MLFMA

FMM

High scalability (parallelization in k-space)High computational cost O(N1.5)

MLFMA

Lowest achievable computational cost O(N logN)Difficult to obtain high scalability



FMM-FFT

Another numerical technique that has gained interest is the FMM-Fast Fourier Transform (FMM-FFT)

The FMM-FFT preserves the natural parallel scaling propensity of the single-level FMM in the spectral (k-space) domain while significantly reducing the computational cost

It is a real alternative for the analysis of very large problems using supercomputers

Previous work

R. Wagner, J. Song, and W. C. Chew, “Montecarlo simulation of electromagnetic scattering from two-dimensional random rough surfaces,” IEEE Trans. Antennas Propag., 45, no. 2, pp. 235-245, Feb. 1997.



FMM-FFT

' '' m

m mm mm B

R T F∉

= ⋅∑

0,0,0 1,0,0 2,0,0 3,0,0

0,0,1 1,0,1 2,0,1 3,0,1

0,0,2 1,0,2 2,0,2 3,0,2

0,0,3 1,0,3 2,0,3 3,0,3

, , ', ', ' ', ', '', ', '

ˆ ˆ ˆ( ) ( ) ( )i j k i i j j k k i j ki j k

R k T k F k− − −= ⋅∑

FMM-FFT

Radiation centers of the groups lie on a rectangular 3D lattice The translation stage in the FMM can be seen as a 3D circular convolution. The acceleration is obtained in terms of the FFT, preserving the natural parallel scaling propensity of the FMM The FMM-FFT combines a low numerical complexity with a high scalability behavior

Computational cost

O(N4/3 log N)



Three stage parallelization strategy

Far contributions: distribution of fields (k-space samples) among processorsNear contributions: distribution of octree groups.

Iterative solver: distribution of unknowns

Benefits

Optimal load balance and data localityMinimizes memory footprint and memory overlappingA single communication step is required at the end of the MVP



Parallel FMM-FFT. High scalability behavior

Currents on a PEC sphere with 10,024,752 unknowns.Solved using from 1 to 1,024 parallel processes (Finis Terraesupercomputer).High scalability behavior.

1. J. M. Taboada, L. Landesa, F. Obelleiro, J. L. Rodriguez, J. M. Bertolo, M. G. Araujo, J. C. Mouriño, and A. Gomez, “High scalability FMM-FFT electromagnetic solver for supercomputer systems”, IEEE Antennas and Propagation Magazine, vol. 51, no. 6, pp. 20-28, Dec. 2009



150 million unknowns

Trade-off for the size of octree groups in very large problems

Small groups imply large 3D FFT matricesLarge groups imply large near-coupling matricesMemory consumption about four times higher than with MLFMA



Nested-FMM-FFT

Nested-FMM-FFT

A refinement of the octree decomposition is appliedThe far contributions are obtained at the coarsest level of the octree using a distributed FMM-FFTThe near contributions are obtained at the finest level using a local shared-memory FMM-FFT inside each computing node



Internaional Awards

Reto computacional: 500 millones de incógnitas

2. J M. G. Araújo, J. M. Taboada, F. Obelleiro, J. M. Bértolo, L- Landesa, J.Rivero, J. L. Rodríguez, “Supercomputer aware approach for the solution of challenging electromagnetic problems”, Progress in Electromagnetics Research, vol. 101, pp. 241-256, 2010.



Contents

MLFMA-FFT

Based on the combination of the best features of MLFMA and FMM-FFTResolution of a problem with more than 1 billion unknowns



Parallel MLFMA-FFT for mixed-memory HPC computers

MLFMA-FFT for mixed memory computers

The MLFMA is the lowest cost algorithm in shared-memoryThe FMM-FFT is highly scalable in distributed memoryThe MLFMA-FFT has shown to be optimum for mixed-memory configurations: high scalability O(N logN) method

3. J. M. Taboada, M. Araújo, J. M. Bértolo, L. Landesa, F. Obelleiro, J. L. Rodríguez, “MLFMA-FFT parallel algorithm for the solution of large-scale problems in electromagnetic (Invited Paper)”, Progr. in Electromagnetics Research, 105, pp. 15-30, 2010



MPI/OpenMP parallel programming

Hybrid parallel programming

Message passing interface (MPI) for distributed computations (FMM-FFT)OpenMP standard for shared-memory computations (MLFMA)Optimal scheme for mixed-memory architectures

OpenMP OpenMP OpenMP

MPI



MLFMA-FFT communications

GROUPS

FIEL

DS




All the communications in the context of the MVP are carried out in only two communication events at the coarsest level L:

forward all-to-all communication: switch from the group driven distribution to the field-driven distribution.

backward all-to-all communication: switches back to the group-driven distribution.

The concentration of communications greatly reduces the degradation due to the network latency idle times.

It also avoids the excessive synchronization of the algorithm




MVP

MLFMA: aggregation, interpolation, near-field translation

Forward communication

FMM-FFT: 3D-FFT far-field translation

Backward communication

MLFMA: anterpolation, disaggregation

Communication of noncontiguous data layout, because of the distributed transposition.

Nonuniform communication volumes, because the size of the actual transferred data to each node differs.

0 1 2 3 4 5 6 7

ML boxes

KL

k-sp

ace

sam

ples

Computing node (rank)

Com

puting

nod

e (r

ank)

0

1

2

3

4

5

6

7




We have addressed these communications by means of the asymmetric collective communication operation MPI::Alltoallw.By properly defining the arrays with the separate send and receive MPI derived datatypes for every node, the complete all-to-all, noncontiguous, nonuniform communications can be completed at once using a single call to the MPI::Alltoallw function.

Pros: the complete management of the communications is left to the MPI library.Cons: the performance of this communication becomes very dependent on the MPI implementation, which may affect the portability.We are using HP-MPI version 2.3.1



December 2009: 620 million unknowns



August 2010: 1 billion unknowns

1,042,977,546 unknowns

NASA Almond at 3 THzICTS HPC resource petition64 nodes of FinisTerraesupercomputer (1024 parallel processors)5 TB of total memory8 iterations GMRES(80)35 hours of executionResidue: 0.023

incE



1 billion unknowns: memory

Geometry, octree and other data

Translation matrices

Near-coupling blocks

Aggregation matrices

GMRES

Lower levels MVP (MLFMA)

3D FFT translation at coarsest level(FMM-FFT)

Maximum available memory



1 billion unknowns: wall-clock time

Translation matrices

Near-coupling blocks

Aggregation matrices

Alltoallv communication

MVP computation and inner com.

Parallel GMRES computation

Geometry input and octree generation (not included in graph): 59 min.



1 billion unknowns: MVP wall-clock time

(166.199 sec.)



Contents

Surface integral-equation (SIE) formulation for generalized media

Left-handed metamaterialsPlasmonic nanoparticles



SIE for plasmonic and artificial materials

Up to now, combining latest algorithmic advances and HPC computing we have been able to analyze EM problems in high-frequency ranges, even reaching the terahertz (THz) or far infrared regionGrowing up in frequency is not only a matter of increasing the number of unknowns.We also must reformulate the algorithms in order to precisely model the specific nature of physics in that frequencies

Objective

To extend their scope of application of SIE methods to emerging fields in nanoscience and nanotechnologyArtificial materials (metamaterials)Plasmonic metallic nanoparticles



Artificial materials:left-handed metamaterials (LHM)

Metamaterials are artificial structures composed of arrangements of scattering elements very small compared to the wavelengthThey provide the possibility for designing artificial media with a controlled photonic responseLeft-handed metamaterials (LHM) provide simultaneously negative permittivity and permeability: negative index of refraction exhibiting exotic properties

LHM applications

Medical imaging, perfect (hyper) lensesCloakingDesigning of antennas, antenna decoupling panels



Artificial materials:left-handed metamaterials (LHM)

Due to its structured composition, realistic numerical simulations of metamaterials can be performed by analyzing the actual lattice of microscopic unit elements

Resolution of very large matrix systemsResonating nature of the microscopic elements: Ill-conditioning and slow convergence

It is very useful for analysis and design purposes to study homogenized LHM in which the constitutive parameters have been retrieved using homogenization techniquesThe integral-equation formulations must be adapted to consider the precise LHM response.

LHM constitutive parameters

( )( )

Re 0

Re 0r

r

ε

µ

<

<? ?k η⇒



Plasmonic metallic nanoparticles:nano-optical antennas

Plasmonic nanoparticles can enhance and direct the spontaneous emission of light in the same way that RF antennas enhance an direct RF emissions.Combining rigorous CEM techniques with the principles of antenna design from radio and microwave technology will enable achieving highly directional nanoantennas for control of light and field enhancement

Wide range of leading-edge applications

Nano-optical communicationsEfficient detection of molecules for biological diagnostics: nano-optical microscopy and spectroscopyQuantum light sources and quantum-information processingHigh-efficiency solar cells



Plasmonic metallic nanoparticles:optical response of metals

The optical response of metals is quite different from that observed in conventional media.This makes it impossible to directly downscale the traditional microwave CEM techniques and antenna designs to the optical regime

Penetration of fields cannot be neglected.Plasmonic response: collective electron oscillations as a response to incident photons.

The integral-equation solutions must be reformulated to consider the precise plasmonic response.

Plasmonic media parameters

( )( )

Re 0

Re 0r

r

ε

µ

<

>? ?k η⇒



We investigate the application of a SIE-Method of Moments (MoM) formulation to the simulation of electromagnetic homogeneous or piecewise homogeneous plasmonic and LHM composite objects

Surface integral-equation (SIE) formulation for the simulation of homogenized LHM

1 1 1( , )R ε µ2 2 2( , )R ε µ

1̂n

2n̂S

1J

2J

1M

2M

Integral equations are obtained from Maxwell’s equations using the equivalence theorems for closed surfaces delimiting the homogeneous penetrable regions.

Equivalence theorem

ˆ

ˆl l l

l l l

= − ×

= ×

M n E

J n H

El, Hl are the total electric fields in medium Rl

inc scatl l l= +E E E

inc scatl l l= +H H H



n̂

S

J M

1 1 1( , )R ε µ2 2 2( , )R ε µ

1̂n

2n̂S

1J

2J

1M

2M

1 1 1( , )R ε µ2 2 2( , )R ε µ

Boundary condition: The tangential field components are continuous across the boundary surface S

⇒

Boundary condition

1 2

1 2

= − == − =

J J JM M M

⇒( ) ( )( ) ( )

1 2tan tan

1 2tan tan

=

=

E E

H H




n̂

S

J M

T-EFIE1T-MFIE1

T-EFIE2T-MFIE2

N-EFIE1N-MFIE1

N-EFIE2N-MFIE2

Tangential/normal combined formulations (JMCFIE)2 2

1 1

1 T-EFIE N-MFIEl l l ll ll

a bη= =

+∑ ∑2 2

1 1N-EFIE T-MFIEl l l l l

l lc dη

= =

− +∑ ∑

JCFIE1 + JCFIE2 =

MCFIE1 + MCFIE2 =

Normal formulations (CNF)2

1N-MFIEl l

lb

=∑

2

1N-EFIEl l

lc

=

−∑

N-MFIE1 + N-MFIE2 =

N-EFIE1 + N-EFIE2 =

Tangential formulations (PMCHWT, CTF)2

1

1 T-EFIEl ll l

aη=

∑2

1T-MFIEl l l

ldη

=∑

T-EFIE1 + T-EFIE2 =

T-MFIE1 + T-MFIE2 =

P. Ylä-Oijala, M. Taskinen, and S. Järvenpää, “Surface integral equation formulations for solving electromagnetic scattering problems with iterative methods,” Radio Sci. 40, RS6002 (2005)




JMCFIE SIE

Method of Moments – Galerkin’s testing method

( )

( ) ( )

( )

( ) ( )

1 2 1 21 2 1 2tan

1 2 1 2tan

1 1 2 2 1 2 1 1 2 2 1 2tan tan

1 21 2

1 2

2 2 1 1 1 2

ˆ ˆ

ˆ ˆ

ˆ( ) ( )12 ˆ( ) ( )

a a b ba a b b

d d d d c c c c

a ab b

d d c c

η η η ηη η η η

η ηη η

+ − − × − × − +

+ + − × − × −

+ − ×+

− × +

n n

n n

n

n

( ) ( ) ( )( ) ( ) ( )

1 1 2 2 1 1 2 2tan tan

1 1 2 2 1 1 2 2tan tan

ˆ

ˆ

inc inc inc inc

inc inc inc inc

a a b b

c c d d

− + × + = − × + + −

E E n H HJM n E E H H

J MJCFIEJCFIE JCFIE

J MMCFIEMCFIE MCFIE

⋅ =

VZ Z JVZ Z M

;n nn

J S= ∈∑J f r

;n nn

M S= ∈∑M f r




LHM: Numerical examples

λ0 radius sphere made of LHMεr = -3, µr = -1 (unmached to free space)RMS error for the bistatic RCS calculation versus Mie series referenceDifferent SIE formulations are tested

Lossless & unmatched case

31

r

rµ= −= −




λ0 radius sphere made of LHMεr = -1, µr = -1 (matched to free space!!)RMS error for the bistatic RCS calculation versus Mie series referenceDifferent SIE formulations are tested

Lossless MATCHED case

11

r

rµ= −= −




All formulations improve their results when losses are included

Lossy MATCHED case

1 0.31

r

r

jµ= − −= −

Lossy unmatched case

3 0.31

r

r

jµ= − −= −



3D LHM planar slab


εr = -4µr = -1

Exterior region:εr = 1µr = 1n = 1η = η0

LHM Slab:εr = -4µr = -1n = -2η = 0.5η0

εr = 1µr = 1

Valid formulations:CTFCNFJMCFIE



3D LHM planar slab


εr = -1µr = -1


LHM Slab:εr = -1µr = -1n = -1η = η0

εr = 1µr = 1

Valid formulations:CTFCNFJMCFIE



3D LHM plane-concave lens






3D LHM perfect lens






Plasmonics: Numerical examples

Gold nanosphere 200 nm radiusIncident plane wave at λ0=550 nmDielectric constant: εr = -8.0 -1.66jJMCFIE with about 2400 unknownsThe computed bistatic RCS shows an excellent agreement with the Mie series referenceAccuracy assessment for different formulations in progress



Plasmonics: nano-optical Yagi-Uda antenna

Yagi-Uda made made of aluminum nanorods5 cylindrical elements 20 nm radiusAntenna optimized for λ0 = 570 nmDielectric constant: εr = -38.0 -10.9jInfinitesimal emitter placed at 4 nm from the lower extreme of the feed elementJMCFIE with about 18000 unknowns (including both J and M unknowns).

[1] T. Taminiau, F.D. Stefani and N.F. van Hulst, “Enhanced directional excitation and emission of single emitters by a nano-optical Yagi-Uda antenna,” Optics Express, vo.16 no.14, July 2008




Antenna directivity in H-plane and E-plane Excellent agreement with the results of Taminiau et al.

H-Plane

E-Plane




To test this antenna in receiving mode (for field enhancement applications) a very thin flat of gap 4nm (about λ/140) was introducedThe total electric field versus the impinging direction simultaneously shows high directivity and field-enhancement capabilities

Número de diapositiva 1Research teamOutlineMethod of Moments.�Computational costMethod of Moments.�RCS of an Airbus A-380 at 1.2 GHzFast Multipole Method (FMM)Multilevel Fast Multipole Algorithm�(MLFMA)Parallelization of codes for mixed-memory HPC supercomputersImportance of scalabilityFMM parallelizationMLFMA parallelizationPros and cons of parallel FMM and MLFMAFMM-FFTFMM-FFTThree stage parallelization strategyParallel FMM-FFT. High scalability behavior150 million unknownsNested-FMM-FFTReto computacional: 500 millones de incógnitasContentsParallel MLFMA-FFT for mixed-memory HPC computersMPI/OpenMP parallel programmingMLFMA-FFT communicationsMLFMA-FFT communicationsMLFMA-FFT communicationsMLFMA-FFT communicationsDecember 2009: 620 million unknowns August 2010: 1 billion unknowns 1 billion unknowns: �memory1 billion unknowns: �wall-clock time1 billion unknowns: �MVP wall-clock time1 billion unknowns: �MVP wall-clock time1 billion unknowns: �MVP wall-clock timeContents SIE for plasmonic and artificial materialsArtificial materials:� left-handed metamaterials (LHM)Artificial materials:� left-handed metamaterials (LHM)Plasmonic metallic nanoparticles:� nano-optical antennasPlasmonic metallic nanoparticles:�optical response of metalsSurface integral-equation (SIE) formulation for the simulation of homogenized LHMSurface integral-equation (SIE) formulation for the simulation of homogenized LHMSurface integral-equation (SIE) formulation for the simulation of homogenized LHMSurface integral-equation (SIE) formulation for the simulation of homogenized LHMLHM: Numerical examplesLHM: Numerical examplesLHM: Numerical examplesLHM: Numerical examplesLHM: Numerical examplesLHM: Numerical examplesLHM: Numerical examplesPlasmonics: Numerical examplesPlasmonics: nano-optical Yagi-Uda antennaPlasmonics: nano-optical Yagi-Uda antennaPlasmonics: nano-optical Yagi-Uda antenna

Documents

Simulation. UESTC Chengdu (China) - unex.estsc.unex.es/~tabo/taboada_chengdu_2011.pdf · 2011 International Workshop on Electromagnetic Theory, Modeling, and Simulation UESTC Chengdu