Approaching the Chiral Limit with Dynamical Overlap Fermionsalgorithm multiplication of DW and Dov overlap solver HMC 3.1 algorithm HMC w/ dynamical overlap quarks on BG/L mult DW:

Approaching the Chiral Limit with DynamicalOverlap Fermions

T. Kaneko for the JLQCD collaboration

1High Energy Accelerator Research Organization (KEK)

2Graduate University for Advanced Studies

“Domain Wall Fermions at Ten Years”, March 15–17, 2007

T.Kaneko Approaching the chiral limit with dynamical overlap fermions

introduction introduction

1.1 introduction

JLQCD: studying lattice QCD using computers at KEK

w/ new supercomputer system (2006 –)Hitachi SR11000, IBM Blue Gene/L (∼ 60 TFLOPS)

⇓large-scale simulations w/ dynamical overlap fermions

⇑computationally expensive ⇐ improvements of algorithm

this talk: algorithmic aspects of production run for Nf =2

lattice action / simulation parameters

our implementation of HMC

production run


setup of simulationlattice actionsimulation parameters

2.1 lattice action

quark action = overlap w/ std. Wilson kernel

Dov =(

m0 +m

2

)

+(

m0 −m

2

)

γ5 sgn[Hw(−m0)], m0 = 1.6

std. Wilson kernel HW ⇒ (near-)zero modes of HW

gauge action = Iwasaki action ⇐ low mode density, locality

extra-fields ⇒ to suppress (near-)zero modesVranas, 2000; RBC, 2002 (DWF); JLQCD, 2006 (ovr)

Wilson fermion ⇒ suppress zero modes

twisted mass ghost ⇒ suppress effects of higher modes

Boltzmann weight ∝ det[HW (−m0)2]

det[HW (−m0)2 + µ2]

extra-fields ⇒ do NOT change continuum limit


setup of simulationlattice actionsimulation parameters

2.2 simulation parameters

Nf =2 QCD

Iwasaki gauge + overlap quark + extra-Wilson (µ=0.2)

β=2.30 ⇒ a ≈ 0.125 fm

163 × 32 lattice ⇒ L≃2 fm

6 sea quark masses ∈ [ms,phys/6, ms,phys]msea = 0.015, 0.025, 0.035, 0.050, 0.070.0.100

focus on Q=0 sector

test runs (500 – 1000 traj.)(β, µ) = (2.30,0.2), (2.45, 0.0), (2.50,0.2), (2.60, 0.0)


algorithmmultiplication of DW and Dov

overlap solverHMC

3.1 algorithm

HMC w/ dynamical overlap quarks on BG/L

mult DW : depends on machine spec.

mult Dov : treatment of sgn[HW]

overlap solver : choice of algorithm, 4D or 5D

HMC : Hasenbusch precond., multiple time scale

multiplication of DW ⇒ assembler code by IBM on BG/L

double FPU instruction of PowerPC 440Ddouble pipelines enable complex number add/mult

use low-level communication APIoverlap computation/communication

⇒ ∼ 3 times faster than our Fortran code



overlap solverHMC

3.2 multiplication of Dov

multiplication of Dov∋sgn[HW]

σ[HW ] ⇒ [λmin, λthrs] ∪ [λthrs, λmax], λthrs =0.045

low mode preconditioningeigenmodes w/ λ ∈ [λmin, λthrs] ⇒ projected out

Zolotarev approx. of sgn[HW ] for λ ∈ [λthrs, λmax]

N = 10 ⇒ accuracy of |1 − sgnHW2| ∼ 10−7

example of λ[HW] (test runs @ a∼0.1 fm, msea∼ms,phys)

w/ extra-Wilson

0 100 200 300HMC trajectory

0

0.01

0.02

0.03

0.04

|λ|

0 300 600hitogram

w/o extra-Wilson

0 100 200HMC trajectory

0

0.01

0.02

0.03

0.04

|λ|

0 50 100hitogram



overlap solverHMC

3.3 4D overlap solver

inner loop:partial fraction form

sgn[HW] ∋NpX

l=1

bl

H2W + c2l−1

multi-shift CG (Frommer et al., 1995)

outer loop:

relaxed CG (Cundy et al., 2004)

D†ov Dov ⇒ CG

× 2 faster than unrelaxed CG

residual |D†ov Dov x − b|

vs # of DW mult (msea =0.015)

0 1e+06 2e+06 3e+06 4e+06# DW mult

1e-15

1e-12

1e-09

1e-06

0.001

1

|Dov

+D

ov x

-b|

2

GMRESCGSUMR(x2)rel. GMRESrel. CGrel. SUMR(x2)



overlap solverHMC


Borici, 2004; Edwards et al.,2005

M5 =(Schur decomposition)⇒ γ5 Dov =Hov as Schur complement

M5 =

0

B

B

B

B

B

B

@

HW −√q2 0

−√q2 HW

√p2

· · · · · ·HW −√

q1 0−√

q1 HW√

p1

0√

p2 · · · 0√

p1 R γ5 + p0 HW

1

C

C

C

C

C

C

A

=

„

A B

C D

«

=

„

1 0

C A−1 1

« „

A 0

0 S

« „

1 A−1 B

0 1

«

S = R γ5 + HW

p0 +X

i

pi

H2W + qi

!

= γ5 (R + γ5 sgn[HW]) ⇒ Hov

(1)T.Kaneko Approaching the chiral limit with dynamical overlap fermions


overlap solverHMC


x = D−1ov b from 5D linear equation

M5

„

χx

«

=

„

0b

«

,

even-odd precond.: implemented

low-mode precond.: not yet...

⇒ need small xmin and large Np

⇔ CPU time ∝ Np

∼4 times faster than 4D CG

residual vs # of DW mult

0 1e+06 2e+06 3e+06 4e+06# DW mult

1e-15

1e-12

1e-09

1e-06

0.001

1

|Dov

+D

ov x

-b|

2

CGrel. CG5D CG



overlap solverHMC

3.4 HMC w/ 4D solver

Hasenbusch preconditioning (Hasenbusch, 2001)

det[Dov(m)2] = det[Dov(m′)2] det

»

Dov(m)2

Dov(m′)2

–

= “PF1” · “PF2”

m′ = 0.2 (msea =0.015, 0.025), 0.4 (msea =0.035 – 0.100)

force (ave,max) at msea =0.015

0 1000.01

0.1

1

10

forc

e

0 100 0 100 0 100 200

PF2

gauge

PF1

ex-Wilsonave.

max.

HMC traj.

PF2 ≪ PF1 ≪ gauge ≈ ex-Wilson

CPU time for force calc (512nodes)

0 1000.01

0.1

1

10

100

time

[sec

]

0 100 0 100 0 100 200

HMC traj.HMC traj.HMC traj.HMC traj.

PF2

gauge

PF1

ex-Wilson

HMC traj.HMC traj.HMC traj.HMC traj.HMC traj.

PF2

gauge

PF1

ex-Wilson


PF2

gauge

PF1

ex-Wilson


PF2

gauge

PF1

ex-Wilson

HMC traj.

PF2 ≫ PF1 ≫ ex-Wilson ≫ gauge



overlap solverHMC


multiple time scale integration

τ = 0.5

3 nested loops:

PF2 : outer-most loop : NMD times / traj.PF1 : intermediate : NMD RPF

gauge,ex-Wilson : inner-most : NMD RPF RG

msea NMD RPF RG m′ PHMC

0.015 9 4 5 0.2 0.890.025 8 4 5 0.2 0.900.035 6 5 6 0.4 0.740.050 6 5 6 0.4 0.790.070 5 5 6 0.4 0.810.100 5 5 6 0.4 0.85



overlap solverHMC


Hasenbusch precond. + multiple time scale

det[Dov(m)2] = det[Dov,5D(m′)2] det

»

Dov,5D(m)2

Dov,5D(m′)2

–

det

»

Dov(m)2

Dov,5D(m)2

–

= “PF1” · “PF2” · “noisy Metropolis test”

sufficiently high “N s” to achieve reasonable PHMC

factor of 2 – 3 faster than HMC w/ 4D solver

msea NMD RPF RG m′ PHMC

0.015 13 6 8 0.2 0.680.025 10 6 8 0.2 0.820.035 10 6 8 0.4 0.870.050 9 6 8 0.4 0.870.070 8 6 8 0.4 0.900.100 7 6 8 0.4 0.91



overlap solverHMC

3.6 reflection / refraction

extra-Wilson fermion

⇒ suppress zero-modes of HW

⇒ switch off reflection/refraction step• reflection/refraction is not rare event!

(at a=0.11 fm w/o extra-Wilson)

⇒ factor of ∼3 faster

w/ extra-Wilson

0 10 20 30 40 50HMC traj.

-0.04

-0.02

0.00

0.02

0.04

λ min

β=2.35, msea=0.090

w/o extra-Wilson

0 10 20 30 40 50HMC traj.

-0.04

-0.02

0.00

0.02

0.04

λ min

β=2.45, msea=0.090


production runparameterspropertiestiming

4.1 production run

10,000 traj. (×τ =0.5) have been accumulated

msea NMD RPF RG m′ traj. PHMC MPS/MV

0.015 9 4 5 0.2 2800 0.89 0.340.025 8 4 5 0.2 5200 0.90 0.400.035 6 5 6 0.4 4600 0.74 0.460.050 6 5 6 0.4 4800 0.79 0.540.070 5 5 6 0.4 4500 0.81 0.600.100 5 5 6 0.4 4600 0.85 0.67

msea NMD RPF RG m′ traj PHMC MPS/MV

0.015 13 6 8 0.2 7200 0.68 0.340.025 10 6 8 0.2 4800 0.82 0.400.035 10 6 8 0.4 5400 0.87 0.460.050 9 6 8 0.4 5200 0.87 0.540.070 8 6 8 0.4 5500 0.90 0.600.100 7 6 8 0.4 5400 0.91 0.67



4.2 basic properties of HMC

area preserving

∆H at msea =0.025

0 1000 2000 3000 4000 5000 6000HMC traj.

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

∆H

a few spikes per O(10, 000)trajectories: Pspike .0.03 %

〈exp[−∆H]〉=1 in all runs

does not need “replay” trick

reversibility

∆U vs ǫ

1e-08 1e-07 1e-06 1e-05 1e-04ε

1.0e-10

1.0e-08

1.0e-06

1.0e-04

∆ U

msea=0.050msea=0.015

∆U =pP |U(τ +1−1)−U(τ)|2/Ndof

ǫ : stop. cond. for MS/overlap solver

∆U . 10−8: comparable toprevious simulations



4.3 effects of low modes of Dov

Ninv,H vs msea

0.00 0.02 0.04 0.06 0.08 0.10 0.12msea

200

400

600

800

1000

Nin

v

msea-0.92

|λov,min| vs msea

0.00 0.02 0.04 0.06 0.08 0.10 0.12msea

10

20

30

1/|λ

min|

msea-0.87

as approaching to ǫ-regime

cost is governed by λov,min rather than msea

too small volume?

MPS L&2.7, exp[−MPS L] ⇒ . 1 – 2% effects on MPS

larger L for msea≪0.015



4.3 timing

# DW mult vs msea

0 0.02 0.04 0.06 0.08 0.1 0.12msea

2e+06

4e+06

6e+06

8e+06

# D

W m

ult /

traj

CPU time [min] on BG/L×10 racksHMC-4D HMC-5D

msea traj. time traj. time0.015 2800 6.1 7200 2.60.025 5200 4.7 4800 2.20.035 4600 3.0 5400 1.50.050 4800 2.6 5200 1.30.070 4500 2.1 5500 1.10.100 4600 2.0 5400 1.0

mild msea dep. of Ninv,H and NMD

⇓

CPU time ∝ 1/m−αsea , w/ α∼0.53

mnaive expectation: Ninv∝1/msea,

NMD∝1/msea

BG/L × 10 racks × 1 month⇒ 4000 traj. at all msea



4.4 autocorrelationτint vs msea

0.00 0.05 0.10msea

0

20

40

60

80

τ

plaquetteNinv

plaquette: local

⇒ small mq dependence

Ninv,H: long range

⇒ rapid increase as mq → 0⇒ may need large statistics

history of Ninv,H

0 1000200

400

600

800

1000

Nin

v,H

0 1000HMC traj.

0 1000

msea=0.015 msea=0.035 msea=0.100msea=0.015 msea=0.035 msea=0.100msea=0.015 msea=0.035 msea=0.100

history of λov,min/〈λov,min〉

0 100

1.0

1.2

1.4

1.6

1.8

|λm

in|/<

λ min>

0 100conf

0 100

msea=0.015 msea=0.035 msea=0.100


summary summary

5. summary

algorithm for JLQCD’s dynamical overlap simulations

Hasenbusch precond. + multiple time scale MD + · · ·

5D solver

extra-Wilson fermion to suppress (near-)zero modes

⇒ cheap approx. for sgn[HW], ⇒ turn off reflection/refraction

effects due to fixed (global) topology (R.Brower et al., 2003)

topological properties (χt,...) ⇒ talks by T-W.Chiu, T.Onogi

Q-dependence of observables ⇐ simulations w/ Q 6=0

suitable for ǫ-regime ⇒ talk by S.Hashimoto

on-going/future plansspectrum/matix elements ⇒ talks by J.Noaki, N.Yamada

simulations of Nf =3 QCDextend to larger volumes


Documents

Approaching the Chiral Limit with Dynamical Overlap Fermionsalgorithm multiplication of DW and Dov overlap solver HMC 3.1 algorithm HMC w/ dynamical overlap quarks on BG/L mult DW: