On Deflations in Extended QR Algorithms

×××××××××××××××××××××××××××××××××××××××××××

×

On Deflations in Extended QRAlgorithms

Thomas Mach Raf Vandebril

Dept. Computer Science, KU Leuven, Belgium

GAMM Workshop Appl. Numer. Linear Algebra09 September 2013Wuppertal, Germany

Thomas Mach, Raf Vandebril, On Deflations in Extended QR Algorithms 1/27

Basics Standard Deflation Aggressive Early Deflation Middle Deflation

Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

QR factorization of the Hessenberg matrixRotation of the form:

[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0×××××

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0×××××

��

QR factorization of the Hessenberg matrix

Rotation of the form:

[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0××××

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0×××

��

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0××

��

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

0×

��

��

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××0

��

��

��


[c s−s c

], |c |2 + |s|2 = 1



Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

=

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

QR factorization of the Hessenberg matrix

Rotation of the form:

[c s−s c

], |c |2 + |s|2 = 1



Extended Hessenberg Matrix

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

There is a relation to extended Krylov subspaces.



Rotators

Fusion:

�� = c1−s1

s1c1

c2−s2

s2c2

= c3−s3

s3c3

= ��

Turnover:

��

��=

Ö

Ö

Ö

Ö

Ö

Ö

Ö

Ö

Ö

= ��

��

Transfer Through Upper Triangular:

��

��Ö

Ö Ö Ö Ö

Ö Ö Ö

Ö Ö=

��

��Ö

Ö Ö Ö Ö

Ö Ö Ö

Ö Ö Ö=

��

��

Ö

Ö

Ö

Ö

Ö

Ö

Ö

Ö

Ö

Ö

� �



Francis’s Implicit Hessenberg QR Algorithm[Francis 1961]

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

1) Compute bulge.

A) Eigenvalues of trailing matrix provide shift.B) x = p(A)e1 = (A− µmI ) · · · (A− µ1I )e1.C) Q0x = γe1.D) Apply Q0.




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

1) Compute bulge.

A) Eigenvalues of trailing matrix provide shift.B) x = p(A)e1 = (A− µ1I )e1.C) Q0x = γe1.D) Apply Q0.




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

��

� �




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×

2) Bulge chasing.

A) Compute zeroing similarity transformation.B) Apply transformation.




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×��

� �

2) Bulge chasing.

A) Compute zeroing similarity transformation.B) Apply transformation.




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×��

� �




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×��

� �




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

×��

� �




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

×××××××




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

×××××××

��




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××

Why Francis’s Algorithm Works [Watkins 1982, ’07, ’08, ’11]

Power method.

Krylov subspaces.

Shifts accelerate convergence.



QR Factorization

QR Factorization



Francis’s Algorithm on the QR Factorization[Vandebril ’11, Watkins, Vandebril ’12]

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �

��

Fusion

Transfer through up. triang.




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Similarity transformation




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

Fusion




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��



Extended QR Algorithm[Vandebril ’11, Watkins, Vandebril ’12]

× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �

��

Fusion





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Move pattern




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

� �


Move pattern




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Turnover




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

��





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

� �





× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

��

Fusion

Move pattern




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��




× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��

Why the Extended QR Algorithm Works

Power method.

Rational Krylov subspaces.

Shifts accelerate convergence.



Standard Deflation

Standard Deflation



Deflation Criterion

× × × × ×× × × ×× × ×× ××

××××

=

��

��

× × × × ×× × × ×× × ×× ××

A) |H(k + 1, k)| ≤ ε ‖H‖F

B) |H(k + 1, k)| ≤ ε (|H(k , k)|+ |H(k + 1, k + 1)|)

A = Gτ(1)Gτ(2) · · ·Gτ(n−1)R

∥∥I − Gτ(i)

∥∥2≤ ε

A = Gτ(1)Gτ(2) · · ·Gτ(i−1)Gτ(i+1) · · ·Gτ(n−1)R



Why is that a Useful Deflation Criterion?

Why is that a useful deflation criterion?



Bauer-Fike Theorem

Bauer-Fike Theorem [Bauer, Fike 1960]

minµ∈Λ(A)

|λ− µ| ≤ κ2(X )‖A− A‖2

‖A− A‖ ≤ ‖Gτ(1) · · ·Gτ(i−1)‖ ‖Gτ(i) − I‖ ‖Gτ(i+1) · · ·Gτ(n−1)‖ ‖R‖≤ ‖Gτ(i) − I‖ ‖R‖≤ ‖Gτ(i) − I‖ ‖A‖

Lemma

Let A be diagonalizable, A = XΛX−1, and λ an eigenvalue of A.Then there exists an eigenvalue µ of A, so that

|λ− µ| ≤ κ2(X )‖Gτ(i) − I‖2‖A‖2.



But we can do better:

Theorem [Eisenstat, Ipsen 1995]

A = A1A2, E = A− A

mini

|λ−µi ||µi | ≤ κ2(X )

∥∥A−11 EA−1

2

∥∥2.

A1 = Gτ(1) · · ·Gτ(i) and A2 = Gτ(i+1) · · ·Gτ(n−1)R

⇒ A−11 EA−1

2 = I − G−1τ(i)

‖I − G−1τ(i)‖2 = ‖G−1

τ(i)(Gτ(i) − I )‖2 ≤ ‖Gτ(i) − I‖2

Lemma

Let A be diagonalizable, A = XΛX−1, and λ an eigenvalue of A.Then there exists an eigenvalue µ of A, so that

|λ−µ||µ| ≤ κ2(X )‖Gτ(i) − I‖2.



Conditioning of One Eigenpair

Theorem [Wilkinson 1965]

Let

µ ∈ Λ(A), Ax = µx , yHA = µyH , be a simple eigenvalue,

E = εF , with ‖F‖2 = 1.

For µ = µ(ε) ∈ Λ(A + E ) with µ(0) = λ, we have

|µ− λ| ≤ ‖E‖2

yH x+O(‖E‖2

2).



Relative Perturbation Estimate

Following the proofs of [Eisenstat, Ipsen 1995] we can proof thefollowing theorem.

Theorem [M., Vandebril ’13]

Let

A be invertible,

µ ∈ Λ(A), Ax = µx , yHA = µyH , be a simple eigenvalue,

E = εF , with ‖F‖2 = 1.

For µ = µ(ε) ∈ Λ(A + E ) with µ(0) = λ, we have

|µ−λ||λ| ≤

‖A−1E‖2

yH x+O(

∥∥A−1E∥∥2

2).

If A = A1A2 and A1 and A2 invertible, then

|µ−λ||λ| ≤

‖A−11 EA−1

2 ‖2

yH x+O

(‖A−1

1 EA−12 ‖2

2

).



Very Good Relative Perturbation Estimate

Corollary [M., Vandebril ’13]

Let A = Gτ(1)Gτ(2) · · ·Gτ(n−1)R and

A = Gτ(1)Gτ(2) · · ·Gτ(i−1)Gτ(i+1) · · ·Gτ(n−1)R. Let further A be

nonsingular, and let λ be a single eigenvalue of A. Then thereexists an eigenvalue µ of A, so that

|µ−λ||λ| ≤

‖I−Gτ(i)‖2

xH y+O

(‖I − Gτ(i)‖2

2

),

with x the right and y the left eigenvector to λ.

The deflation criterion∥∥I − Gτ(i)

∥∥2≤ ε is very good.



Aggressive Early Deflation (AED)[Braman, Byers, Mathias ’02]

Aggressive Early Deflation



Aggressive Early Deflation (AED)[Braman, Byers, Mathias ’02]

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

×××××××××

A

⇒

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

××××××⊗⊗⊗

A′ = QHAQ

VH

× × × ×× × ×× ××

V



AED for Extended QR Algorithms

��

��

��

��

��

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ×××××

×××




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

VH V




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

��

��

��

��




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

�� ×

�� × ×

�� ×

�� ××��




× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Wedge-to-Broadhead Transformation

��

�� =��

��

��

��

��

��=

��

��

��

��

=

��

�

�

�

�

�

�

��

��

Test for deflations.



Restore Original Pattern

×××××××××××××××××××××

⇒×××××××××××××××××××××

⇒×××××××××××××××××××××

⇒

×××××××××××××××××××××

⇒×××××××××××××××××××××

⇒×××××××××××××××××××××

⇒

×××××××××××××××××××××

⇒×××××××××××××××××××××

⇒×××××××××××××××××××××



The effect of aggressive early deflation for n = 500

1020

3040

50

020

4060

80100

0

200

400

*

*

# shifts#

deflation window

run

tim

ein

s

aggressive early deflationstandard deflation only

best without early deflation best with early deflation savingsn sh t #it·sh sh m t #it·sh

100 4 6.00 s 215 10 18 5.62 s 223 6% time100 2 6.42 s 198 2 28 8.60 s 163 18% iterations500 6 148.49 s 803 20 20 100.45 s 727 32% time500 2 161.38 s 732 2 70 314.85 s 391 46% iterations



The effect of aggressive early deflation for n = 500

1020

3040

50

020

4060

80100

0

200

400

*

*

# shifts#

deflation window

run

tim

ein

s

aggressive early deflationstandard deflation only

best without early deflation best with early deflation savingsn sh t #it·sh sh m t #it·sh

100 4 6.00 s 215 10 18 5.62 s 223 6% time100 2 6.42 s 198 2 28 8.60 s 163 18% iterations500 6 148.49 s 803 20 20 100.45 s 727 32% time500 2 161.38 s 732 2 70 314.85 s 391 46% iterations



Middle Deflation

Middle Deflation



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××

V VH V VH



Middle Deflation

��

�� ×

× ×

��

��×××

��

��

× ××

��

��

× ××

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Middle Deflation

× × × × × × × × × ×× × × × × × × × ×× × × × × × × ×× × × × × × ×× × × × × ×× × × × ×× × × ×× × ×× ××



Predicting Middle Deflations

One needs to define a deflation window that most likely leads to amiddle deflation.

g∏i=f

|si | < ε2

For random matrices a successful middle deflation window of size0.75 · n × 0.75 · n is found after a few iterations.

A ∈ C1500×1500, random, three multishift steps (150 shifts),deflation window 363:1026, leads to middle deflation

The corresponding deflation windows for other test-matrices(tols2000, pde2961) have not been successful.



Predicting Middle Deflations

One needs to define a deflation window that most likely leads to amiddle deflation.

g∏i=f

|si | < ε2

For random matrices a successful middle deflation window of size0.75 · n × 0.75 · n is found after a few iterations.

A ∈ C1500×1500, random, three multishift steps (150 shifts),deflation window 363:1026, leads to middle deflation

The corresponding deflation windows for other test-matrices(tols2000, pde2961) have not been successful.



Conclusions

We have investigated deflations for extended QR algorithms.

Deflating almost diagonal rotations preserves absolute andrelative accuracy also with respect to the conditioning ofsingle eigenvalues.

We have seen aggressive early deflation in extended QRalgorithms.

These ideas can be used for middle deflations in special cases.



Acknowledgments

The research was partially supported by

the Research Council KU Leuven

CREA-13-012 Can Unconventional Eigenvalue AlgorithmsSupersede the State of the Art (CUASSA),F+/13/020 Exploiting Unconventional QR-Algorithms for Fastand Accurate Computations of Roots of Polynomials,OT/11/055 Spectral Properties of Perturbed Normal Matricesand their Applications,CoE EF/05/006 Optimization in Engineering (OPTEC),

DFG research stipend MA 5852/1-1,

Fund for Scientific Research–Flanders (Belgium) project G034212N,Reestablishing Smoothness for Matrix Manifold Optimization viaResolution of Singularities,

Interuniversity Attraction Poles Programme, initiated by the BelgianState, Science Policy Office, Belgian Network DYSCO – DynamicalSystems, Control, and Optimization.


Technology

On Deflations in Extended QR Algorithms