Radar Problem Real-Life Problem - TAU

1

www.abakus.computing.technology

II EE EE EE AA PP Distinguished Lecturer

LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm

Fast Algorithms and Parallel Computing: Solution of Extremely Large Real-Life Problems

in Electromagnetics

Levent Gürel CEO, ABAKUS Computing Technologies

Professor Emeritus, Bilkent University, Turkey Founder, Computational Electromagnetics Research Center (BiLCEM)

Adjunct Professor, Dept. of ECE, University of Illinois at Urbana-Champaign




!"#$""%&'

(")**+',+*-%.%/*"'

,0*#*"12'3+4&#%5&'

6$#%7%#$+1%5&'

8%)%+'94&#$7&'

:-/2%5''(7%.1".''94&#$7&'

;1*5*.12%5'9#+<2#<+$&'

=8;3>'

?1+$5$&&'3*77<"12%/*"&'

@ABC8%".$'31+2<1#&'

!"#$%&'(")*$"+,-./#01"2+3)4-.5$"/3)6,)7'8"-"1,)9-":;"1+'"3)



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Radar Problem



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Real-Life Problem

Electromagnetic Modeling of Microcircuits

2



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Real-Life Problem

Nano-Optical Imaging Systems




20 GHz

30 GHz

Photonic crystals

!"#$%&'()*+),-'&'.*/-.)0/+1&'$()

Ö. Ergül, T. Malas, and L. Gürel, Analysis of dielectric photonic-crystal problems with MLFMA and Schur- complement preconditioners, J. Lightwave Technol., vol. 29, no. 6, pp. 888–897, Mar. 2011.



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Radiation into Living Organisms




Rx Antenna Rx Antenna

Rx Antenna

Rx Antenna

Tx Antenna

Tx Antenna

Tx Antenna

Tx Antenna

Microwave Imaging

3




Double-Ridged Horn

Fractal Spiral

Log-Periodic

Vivaldi

Conical-Corrugated Horn

Archimedian

Patch+SRRs

Conical Spiral

Patch

Antennas



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Mobile Device Antennas




Satellite Antennas



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Antennas Mounted on Platforms

•! Interaction of multiple antennas •! Characteristics of mounted antennas (different from isolated antennas) •! Optimization of the placement of the antennas

4



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm 2-$3&#4+5)!56-/+5$'5*)




Maxwell s Equations

!"E(r ,t) =#

$$t

B(r ,t)

!"H(r ,t) =

##t

D(r ,t)+J (r ,t)

!"B(r ,t) = 0

!"D(r ,t) = !(r ,t)




Surface Integral Equations

CFIE = !EFIE + (1! !)MFIE

•! Electric-Field Integral Equation (EFIE):

•! Magnetic-Field Integral Equation (MFIE):

•! Combined-Field Integral Equation (CFIE):

•! Hybrid-Field Integral Equation (HFIE):

HFIE = !(r)EFIE + 1! !(r)"

#$%&'MFIE

J(r)! n̂(r)" d #r J( #r )" #$ g r, #r( )#S% = n̂(r)"H

inc(r)

!t̂(r) " ik d #r I ! $#$

k2

%

&''''

(

)****g r, #r( ) "J( #r )

#S+ =

1!t̂(r) "E inc(r)



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Geometry Discretization

Stealth Antennas

Airborne

5



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Discretization

ZmnE = dr tm(r)

Sm

! " d #r G r, #r( ) "bn( #r )Sn

!

ZmnM = dr tm(r)

Sm

! "bn(r)# dr tm(r)Sm

! " n̂ $ d %r bn( %r )$ %& g r, %r( )Sn

!

Zmnan =n=1

N

! vm, m = 1,2,…N

•! Matrix equations:

•! Matrix elements:

Zmn

C = !ZmnE + (1! !)

ik

ZmnM

Basis functions

Testing functions

J(r) = an bn(r)

n=1

N

!

Number of unknowns

Z

mn

H = !mZ

mn

E + (1! !m

)i

kZ

mn

M



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Matrix Elements…

…are electromagnetic interactions

ZmnE = dr tm(r)

Sm

! " d #r G r, #r( ) "bn( #r )Sn

!

ZmnE an =

n=1

N

! vmE , m = 1,2,…N

Basis functions

Testing functions



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Geometry Discretization

Modeling with millions of triangles.

Mesh Size: !/10



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Matrix Equation

System of Linear Equations

A!x=b

Z !a=v

6



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Iterative Solutions

Z !a = v

x

Z !x = y y

M ! z = w

w

z

•! Large matrix equations

a

•! CG and CGS •! BiCG and BiCGStab •! QMR and TFQMR •! GMRES •! LSQR

Parallel Multilevel Fast Multipole Algorithm (MLFMA)

Matrix-Vector Multiplications Iterative

Algorithm

•! BDP •! NFP •! Filtered NFP •! ILU and ILUT •! SAI •! Nested PCs

Preconditioner



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Fast and Faster Solvers

MOM FMM MLFMA

Parallel

MLFMA

1.5 GHz

2 GB hafıza

8000 bilinmeyen

3 GHz

2 GB hafıza

30.000 bilinmeyen

6 GHz

1,4 GB hafıza

130.000 bilinmeyen

12 GHz

650 MB hafıza

520.000 bilinmeyen

8 bilgisayar

0,3m yarıçaplı küre için

frequency

Higher frequency

Larger problems



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Fast Multipole Method

Aggregation

Translation

Disaggregation

Cluster of basis functions

F !G

(k̂) = Fn !G

(k̂) an

n" !G#

F

GT(k̂) = T

L(k,D)

!G "N (G)# F !G

(k̂)

Z

mna

nn=1

N

! =14!

d 2k̂ FmGC (k̂) "F

GT(k̂)#

Cluster of testing functions

Complexity: O(N3/2)



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm <;$2$"="$)9#3,)<;$2>.$")6$0.-',?/

7




7#8-#*'8))+/)-5.+$-59):'&8()

O(N logN )!!);+$%&'"-*<=)

>/'')(*/3.*3/')7'.3/(-6').&3(*'/-59)

;&3(*'/()

?3&4&'6'&)@#(*)?3&4%+&')A&9+/-*B$)




Reduced-Complexity Solutions

Astrophysics (Gravitational Force) Acoustics

Electrodynamics (Helmholtz s Equation)

Molecular Dynamics

Quantum Mechanics (Schroedinger s Equation)

Electrostatics (Laplace s Equation)

Fast Multipole Methods

Structural Mechanics Fluid Dynamics

Heat Equation



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Books



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Parallel Computing

NODES

NETWORK SWITCH

SERVERCONNECTED TO INTERNET

SLAVE HOST

Constructing a Parallel Computer Cluster

8



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Parallel Computers



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Parallel Computing

Constructing Parallel Computer Clusters

8 nodes

Per node: Two Intel Xeon 5345 2.33 GHz Quad-Core Processors

Total of 64 cores

Per node: 32 GB DDR2-667 533 MHz RAM

Total of 256 GB of RAM

Network: Infiniband

17 nodes

Per node: Two Intel Xeon 5472 3 GHz Quad-Core Processors

Total of 136 cores

Per node: 32 GB DDR2-667 533 MHz RAM

Total of 544 GB of RAM

Network: Infiniband

136-Core Cluster 64-Core Cluster

Not in www.top500.org!!!




What is the Main Source of Efficiency?

N Unknowns

O(N3) Gaussian Elimination

O(N2) Iterative MOM

(MVM)

O(N3/2) Single-Level

FMM

O(N logN) Multi-Level FMM

1000 1 s 2 s 4 s 8 s

106 32 years 23 days 35 h 7 h

107 32 K years 6.3 years 46 days 89 h

108 32 M years 630 years 4 years 46 days

109 32 G years 63 K years 127 years 1.5 years

(555 days)



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm 53 Million Unknowns

Sphere with radius of 120! and diameter of 240! November 2007

120!

53,112,384 Unknowns

9




Sphere with radius of 150! and diameter of 300! December 2007

150!

69,177,600 Unknowns




Sphere with radius of 150! and diameter of 300!

January 2008

150!

85,148,160 Unknowns

Scattering results are compared to analytical values to demonstrate the accuracy of the numerical solution.




How Large are Large Matrix Equations?

Each element of the matrix is a complex number with real and imaginary parts

If we could fit each element of the matrix in a square of 1 in by 1 in…

1 in

1 in 7.654321E+02 + j 1.234567E-03



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm 85 Million " 85 Million Matrix

10



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Intel Pamphlet on the World Record

Available at www.cem.bilkent.edu.tr





August 2008

135,164,928 Unknowns


180!





August 2008



190!





September 2008



210!

11



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Parallelization

Work load in the tree structure

Field samples

Clusters www.abakus.computing.technology







Field Sampling

Coarse Sampled Data Big Data!

Fine Sampled Data





Field samples

Clusters

12



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Partitioning of the Tree Structure In the parallelization of MLFMA, the main work is partitioning the tree structure

Tree structure

Field samples

Clusters

Simple Partitioning

-! Clusters are distributed in all levels

-! Inefficient for high levels (poor load-balance)



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Hybrid Partitioning of the Tree Structure

(Chew et al.) Hybrid Partitioning (improves load-balance for higher levels)

Distributed levels:

Shared levels: Translations are communication-free Aggregations & disaggregations require communications

Translations require communications Aggregations & disaggregations are communication-free

Problem: There are some levels, where neither distributing fields nor clusters is efficient



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Partitioning of the Tree Structure

Hierarchical Partitioning

Fields and clusters are “perfectly” distributed




Partitioning of Multilevel Tree •! Hierarchical partitioning

Fields and clusters are “perfectly” distributed

•! Define intermediate levels

Level 2

Level 3

Level 2.5

Modify partitioning

Aggregation/Disaggregation

12

34

56

78

1

2

3

4

5

6

7

8

1324

5768

13



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Timeline of Large Problems





December 2009



260!





December 2009



280!





September 2010



340!

14





2013

670 Million Unknowns Scattering results are compared to analytical values to demonstrate the accuracy of the numerical

solution.

340!

340!





2014


880!



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm 1.1 Billion Unknowns


2014


1000!



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Metamaterials

Split-Ring Resonators

15




SRR Walls dB

dB dB

1 Layer

4 Layer 2 Layer

Pow

er T

rans

mis

sion

(dB

)

Frequency (GHz)

Meta Property: Stop band in frequency!

No Pass

Pass



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Special Configurations

Closed-Ring Resonators (CRRs)

Thin Wire Array (TWA)

All pass! No pass!



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Composite Metamaterial (CMM)

Split-Ring Resonators (SRRs)

Thin Wire Array (TWA)




CMM Walls dB

dB dB

1 Layer

4 Layer 2 Layer

Pow

er T

rans

mis

sion

(dB

)

Frequency (GHz)

Meta Property: Pass band in frequency!

No Pass

Pass

16



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Large MM Problem

20 x 51 x 29 SRR Block

2,425,560 Unknowns

MLFMA parallelized into 64 process on a cluster of Intel-Xeon processors connected via Infiniband network

85 GHz

100 GHz



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm @*ABC<6!DEAF)

A6#-&#1&')C/+$)DDDE#1#F3(E.+$%3459E*'.B5+&+9<)

2.#G'/-59)C/+$)(%B'/')H/#8-3(=)IJ! K)LMJ!)!N'1K1#('8)#%%&-.#4+5=)O%&+#8)*B').+$%3*#4+5#&)/'(3&*()#58)9'*)*B')'//+/)D-*B)/'(%'.*)*+)#5#&<4.#&)?-'K('/-'()(+&34+5(E)

2.#G'/-59)C/+$)PA2A)A&$+58)H(-Q'=)RM! K)STSM!)!N'1K1#('8)#%%&-.#4+5=)O%&+#8)*B').+$%3*#4+5#&)/'(3&*()#58)9'*)*B')'//+/)D-*B)/'(%'.*)*+)+3/)/'(3&*(E))



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm G*@%@6H*7)@*ABC<6!DEAF)IJJ&)

N'1(-*')358'/).+5(*/3.4+5U)N-&&)1')#6#-&#1&')C/+$)

DDDE.'$E1-&F'5*E'83E*/)




64/26

Red Blood Cells (RBCs) Ordinary (Healthy)

Red Blood Cell Macrocyte

(Macrocytosis) Microcyte

(Microcytosis)

Spherocyte (Spherocytosis) Sickle Cell

(Sickle Cell Anemia)

17



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Flow Cytometry

Blood sample

Forward scattering

Side scattering Backscattering

Source: http://nirfriedmanlab.blogspot.in/2010_04_01_archive.html



LLeevveenntt Gürel AACCCC,, TTeell AAvviivv UUnniivveerrssiittyy AAnntteennnnaass MMiinnii--SSyymmppoossiiuumm Scattering from and Imaging of

Red Blood Cells Electromagnetic fields




Forward Scattering

No Detection

Detection of an Extra-Ordinary Cell





Forward Scattering

No Detection



An extra-ordinary cell may have normal

SCS values

An extra-ordinary cell can be detected for some orientations

An extra-ordinary cell can be detected for all orientations

18




Decision Chart




SOLUTION OF LARGE

COMPUTATIONAL ELECTROMAGNETICS

PROBLEMS

Mathematics Integral Equations Numerical Analysis

Linear Algebra Iterative Solvers Preconditioners

Computer Engineering Parallel Architectures Multicore Processors

Computer Science Fast Algorithms

Parallel Programming

Geometry Modelling 3-D CAD Modelling

Meshing

Physics Electromagnetic Theory Radiation and Scattering

Multi-Disciplinary Approach

Documents

Radar Problem Real-Life Problem - TAU