LGT Software & Multigrid · and James Osborn: VERY EARLY TESTS on Wilson Lattice from Kostas. • Looks promising with much development • Chronological Preconditioner • Retuning

LGT

Software &

Multigrid

Rich Brow

erBoston U

niversity

Lattice QC

D C

omputational Science W

orkshopO

ak Ridge

April 30, 2013

Tuesday, April 30, 13

“Algorithm

s” for “solving” Quantum

Field Theory

Path Int = Integral[ exp[ - A

ction]

Feynman

Diagram

sRenorm

alizationG

roup

PDE/FEM

Schwartz

Schurs

Wilson

Lattice QC

D

‘tHooft

Dim

Reg

Multi-

Grid

Real-Space

GR

OPE

AdS/C

FTTw

istersBootstrap

DD

Dom

ainW

allW

ilsonFlow


USQ

CD

Software Stack Stack

On line distribution: http://usqcd.jlab.org/usqcd-softw

are/

The application codes C

hroma/C

PS/MILC

and a new Q

DP LU

A code base provide a rich set of tools.

QLA

/perl = 23000 files

!"#$%&'#&("')

!"#$%&'()

*+

!""#$!

%&'()#*** Chrom

a = 4856 files

CP

S = 1749 files

MILC

= 2300 files

QU

DA

/python = 221 files Level 3

Level 4

Level 2

Level

!"#$%&

!'(&#)*(+,&!(')-.*(&}

Apps Softw

arenot

SciDA

C funded


SciDAC

LGT contributors

•ANL: Jam

es Osborn, M

eifeng Lin, Heechang Na, (George T. Flem

ing)•

BNL: Frithjof Karsch, Chulwoo Jung, Hyung-Jin Kim,Yu M

aezawa•

Columbia: Robert M

awhinney, Hantao Yin•

FNAL: James Sim

one, Alexei Strelchenko, Don Holmgren, Paul M

ackenzie•

JLab: Robert Edwards, Balint Joo, Jie Chen, Frank Winter, Chip W

atson•

W&M

/UNC: Kostas Orginos, Andreas Stathopoulos, Rob Fowler (SUPER)

•LLNL: Pavlos Vranas, Chris Schroeder, Rob Faulgot (FASTm

ath) •

NVIDIA: Ron Babich, Mike Clark

•Arizona: Doug Toussaint, Alexei Bazavov

•Indiana/NCSA: Steve G

ottlieb, Ran Zhou

•Utah: Carleton DeTar, Justin Foley

•BU: Richard Brower, M

ichael Cheng, Oliver W

itzel•

MIT: Pochinsky Andrew, John Negele,

•Syracuse: Sim

on Catterall, (David Schaich in fall)•

Washington: M

artin Savage, Saul Cohen•

Others: Peter Boyle, Jim

Hetrick, Massim

o Di Pierro, Patrick Dreher, et al•

“Team of Rivals” (apologies to contributors and projects *NO

T* mentioned in 6 slides!)


Highest Priority is m

oving to 3 new architecture!

(May you live in I

nteresting Times!)


•!ab initio C

hemistry!

1."1930+50 = 1980!

2."0.1 flops !

10 Mflops!

3."G

aussian Basis functions!

•!ab initio Q

CD!

1."1980 + 50 = 2030?*!

2."10 M

flops ! 1000 Tflops!

3."C

lever Collective Variable?!

vs!

Lattice Field Theory is just now

meeting the W

ilson criterion for relevance to Experim

ental Physics!


We are just now

able to resolve the pion scale!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

"#$%!&

'()

!*+#,)

*!#

()!-

#./.$

0!.$

!,.$)1

23#(4

*!&

#**)*1!!53

6*+7/!8

!9:!;

:!<==:!!<

>==:!?

<@=:!9

=====!"

)AB

C,)

+/('

&#0$)/.*

&!5D

('/'

$E$

3+,)

'$!*D,.//.$

0:!0

E9B

F.$

6.$

0!)

$)(0

%!'

G!$3+,)

.!59H9

!")I!G'

(!6)3/)

('$B

J)A!!K

/('$0!L

#30)!F

K"

!5$)#(!+

'$G'

(&#,B!6

%$#&

.+*!G'

(!+'&

D'*./)

!M.0

0*!

New

Algorithms: C

rucial for both New

Physics & Architectures

a(lattice)

⌧1/M

proton

⌧1/m

⇡

⌧L(box)

0.06ferm

i⌧

0.2ferm

i⌧

1.4ferm

i⌧

6.0ferm

i

=)L=

O(100)orMinimum

LatticeVolume

100

4!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

a(lattice)

⌧1/M

proton

/M

proton

/M

⌧1/m

⇡

⌧L(box)

0.06ferm

i⌧

0.2ferm

i⌧

1.4ferm

i⌧

6.0ferm

i

=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=)=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!L

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!L

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!(100)orMinimum

LatticeVolume

100

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!(100)orMinimum

LatticeVolume

100

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!4!


* “MG

is always the Future”: Joke from

JLab 2008

** “The future has arrived

!”: Message at Oak

Ridge 2013

•History Lessons (1989-1992) *

-Cause of early failure for 20 years!

•M

odern Era (2008-2013)

-5 years to put into production the Q

CD MG

Solver for Wilson-clover

•Future** (2013-2018)

-Dom

ain Wall & Staggered Solvers, HM

C evolution, etc

-Adaptation to heterogeneous architectures, etc.

Multigrid: C

ase History in A

lgorithm D

evelopment


Past History


QC

D M

G attem

pts in 1990’sS

ee Thomas K

alkretuer hep-lat/9409008 review

on “MG

Methods

for Propagators in LG

T”.

Israel: Ben-Av, M

. Harm

atz, P.G

. Lauwers &

S.S

olomon

Boston: B

rower, E

dwards,

Rebbi &

Vicari

Am

sterdam: A

. Hulsebos,

J Sm

it J. C. Vick

Am

sterdam: A

. Hulsebos,

J Sm

it J. C. Vick


QC

D M

G “failure” in 1990’s:


2-D W

ilson Dirac O

perator


Universal “failure” of C

ritical Slowing dow

n:

⌧=

F(m

l�)

�=

3(cross),10(

plus),100(

squares)

Gauss-Jacobi(Diamond),CG

(circle),

Multi-Grid3levels(square

star)


!"##$%%&'&()*+",$%&-.&/

0&)11$2

31%&*4&56678%&9&:;<=&

•K3++)**1

•"

#.$

/#.$

!L#30)!N$

I#(.#

$+)!

•"

#.$

/#.$

)6!L

#&

&#E;

!M)(&

./.+./%

•O'+#,!#

6#D/.I

)!7

,'+4.$

0!5-

./P!Q

('R)

+/.I

)!"

LB

•Q#(/.#

,!*3++)**!-

./P!5S

LB!#

/!-)#4!+'3D,.$

0

•T#.,)

6!#

/!*/('

$0!+'3D,.$

0H!

•L#,)

(4.$

!G'(&

corseoperator:D

c=

RDP

Boris Grigoryevich G

alerkin (Russian: !"#$%& '#$("%#)*+$, '-./#0$1, surnam

e more

accurately romanized as G

alyorkin; March

42[O.S. February 20, 1871]218712– July 12, 1945),


Why D

idn’t It work?


!"#$%&'()**&'+,,-$$.&&&/

00*"-1&2)%34536$",$&&78**)98#)%"8:

First Success: Applied M

ath/Physics Collaboration C

ollaboration

•! Saul C

ohen • S

aul Cohen

•!INT S

eattle •! S

aul Cohen

•M

ichael Cheng

• NV

IDIA

•O

liver Witzel

•! Ron B

abich

• MIT• A

ndrew Pochinsky

• Mike C

lark

• Saul C

ohen •! S

aul Cohen

Chris Schroeder

•! Ron B

abich


Present Break Through


Adaptive Sm

ooth Aggregation A

lgebraic Multigrid

smoothing

Fine Grid

Sm

aller Coarse G

rid

restriction

prolongation (interpolation)

The Multigrid

V-cycle

Spilt the vector space into near null space S and the com

plement S

!!

D: S !

0!


What is the N

ew Idea?

•M

ath Speak: A Schur/Schwarzian DD splitting of the vector space:-

How do you spit the space into Fine vs Coarse Space?

-Classical M

G vs Adaptive M

G

(see Front cover of Strang’s Undergraduate MIT m

ath text!)

•In Physics Speak: The W

ilsonian Renormalization G

roups:-

How to separate UV (short scales) from IR (long scales)

-Conform

al (Scale Inv) vs Non-perturbative RG

But P†P = 1cc so Ker(P) = 0!

S!!

S span(P)!

ker(P†)!

P! P†!

span(P†)!

UV!IR

!

fine space!K

arl Herm

ann Am

andus Schw

arz (25 January 1843 – 30 N

ovember 1921)


Devils in the D

etails!•

To turn the new Mathem

atical idea into a competitive M

G

solver took a lot of work!-

First in principle demonstration of coarsening used the norm

al equations: no dependence on the m

ass (lowest e.v.) but was not com

petitive.

-Needed to go to Red/Black precondition non-Herm

itian operator -- not conventional M

G.

-O

(25) near null vector on 4^4 blocks plus chirality.

-G

CR smoother etc etc.

•See:

•Code is available: Docum

entation this sumer?

Adaptive multigrid algorithm

for the lattice Wilson-Dirac operator, R. Babich, J.

Brannick, R. C. Brower, M. A. Clark, T. A. Manteuffel, S. F. McCormick, J. C. Osborn and C. Rebbi, Phys. Rev. Lett. 105 (2010) 201602.


Adaptive Sm

ooth Aggregation A

lgebraic Multigrid

SciDA

C com

pliant production code on BG/P


Good N

ews/B

ad New

s

More D

ata: Should Save M

G

projectors with

lattice

Actually M

G error

is smaller at fixedR

esidual



Near Future Projects


What about D

omain W

all and Staggered?

•Dom

ain Wall attem

pt Saul Cohen, M. Clark and J. O

sborn -

Must use Norm

al Equations (so far).

-But Blocking elim

inate 5th dimension!

-The algorithm

is still not tested.

-New effort with O

liver Witzel and Andrew Pochinsky in Q

LUA

•Staggered. -

Red/Black precondition is the Normal Eq.

-Theoretical straight forward

-BUT “double” is a 4 fold increase in near null space.

-M

ay have to have blocks preserving the tastes?

-Carleton ?

Multigrid A

lgorithms for D

omain-W

all Fermions, S. D. Cohen, R. C. Brower,

M. A. Clark and J. C. Osborn, PoS LATTICE 2011, (2011) 030


Non N

ormal !

Non H

ermitian!

Non Pos. D

ef.!

Non-norm

alnon-H

ermitian

Non Pos. D

ef.

!"#$%&''()*+,-'


What about M

ultigrid HM

C?

New

Project with M

eifeng Lin and James O

sborn: VERY

EA

RLY

TEST

S on Wilson Lattice from

Kostas.

• Looks promising w

ith much developm

ent• C

hronological Preconditioner• R

etuning HM

C algorithm

• Bigger Lattices• N

ote: Lattice will be stored W

ITH

the near NU

LL vector so there is no set up cost to M

G in analysis


Herterogeneous A

rchitecture•

Problem: W

ilson Clover for Light Quark is FASTER on the

CPU than using the QUDA solver on G

PUs!

•Solution put M

G on G

PU of course !!!"#$%&'%(#%)*+,-*+%./%0%10-$")%"1%0$%2*0#$%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 3456678%

%%%%9:;

%%%345678%%%%%%%%%%%%%%%%%%<

9%%%3

45678%

="'%>0.&-?%%%%%%%%0'+%%<

&@*%!20)@%%%%%%%0'+%%%%A-&B

C!%

%%%%%%0'+%%%%%D

E&+&0%

(now with Mike Clark at NVIDIA and M

ichael Cheng on NSF grant)Tuesday, April 30, 13

Rapid Prototyping and Tuning? •

HYPER in FastMath with Pavlos Vranas, Chris Schroeder,

Rob Faulgot (FASTmath)

•Q

LUA = QCD + LUA is FUN

-https://usqcd.lns.m

it.edu/w/index.php/Q

LUA

_tutorials

Integration of HY

PRE and Q

lua is well underw

ay. Rob Falgout, C

hristopher Schroeder and Andrew

Pochinsky have completed an overall design of a H

YPR

E/USQ

CD

interface (HQ

L) and begun its

implem

entation. RF is largely finished extending H

YPR

E to handle more than 3 dim

ensions and fully

expects to finish implem

enting complex num

bers on schedule. CS and R

F are making progress on

the implem

entation of the HQ

L interface, and RF and A

P are proceeding with the H

QL-Q

lua

interface. AP is finishing extending Q

lua to handle data types and procedures required to support

HQ

L.


FUEL: Jam

es Osborn for BSM

HM

C developm

ent


RxSherical Lattice: R

x S3•

Radial Quantization requires “spatial” spheres!

-Conform

al Fix Points for BSM theories

-(or partial wave expansion & Scattering Length?)

-Need Finite Elem

ents Method to do 3d Ising on curves space!

exact icosahedral symm

etryTuesday, April 30, 13

Mu

ch m

ore A

lgorith

mic D

evelop

is a critica

l investm

ent

•* M

G D

omain W

all &Staggered

•* M

ulti-grid/HM

C Evolution

•* M

ulti-grid QU

DA

on GPU

s

•R

esearch Area for “real” D

D

• M

ultiGrid/D

D “deflation”

• *W

ilson Flow

•D

isconnected Diagram

s

•EigC

G D

eflation & A

ll mode averaging

(48^3 96 24: Big Data, 600 ev

7.2TBytes)

•M

ulti-quark contraction codes

•*R

adial Quantization: C

onformal T

heory

•LO

Ts M

OR

E to DO

.

•A

lgorithmic advances are difficult and tim

e consuming

Need separate D

evelopment and O

ptimization Softw

are Environment

Need M

odularity to move new

algorithm into A

pplication Codes.

Conclusions: “T

he Sun is But a Morning Star*”!

* Henry

David

#oreau

in Conclusion

% Walden

, 1845


Documents

LGT Software & Multigrid · and James Osborn: VERY EARLY TESTS on Wilson Lattice from Kostas. • Looks promising with much development • Chronological Preconditioner • Retuning