Look-ahead in Bi-CGSTAB and other product methods for ...reichel/courses/intr.num.comp.2/spring14/lecture13/... · BIT 35 (1995), 169-201 . LOOK-AHEAD IN BI-CGSTAB AND OTHER PRODUCT

BIT 35 (1995), 169-201 .

LOOK-AHEAD IN BI-CGSTAB AND OTHERPRODUCT METHODS FOR LINEAR SYSTEMS

C . BREZINSKI' and M . REDIVO-ZAGLIA2

2Laboratoire d'Analyse Numerique et d'Optimisation, UFR IEEA - M3Universite des Sciences et Technologies de Lille, F-59655 Villeneuve d'Ascq

Cedex, France . email: brezinsk@omega .univ-lillel .fr2 Dipartimento di Elettronica e Informatica, Universite degli Studi di Padovavia Gradenigo 6/a, 1-35131 Padova, Italy . e mail: elen@elettl .dei .unipd .i t

Abstract .

The Lanczos method for solving Ax = b consists in constructing the sequence ofvectors xk such that rk = b - Axk = Pk(A)ro where Pk is the orthogonal polynomialof degree at most k with respect to the linear functional c whose moments are c(~') _ci = (y, A'ro) .

In this paper we discuss how to avoid breakdown and near-breakdown in a wholeclass of methods defined by rk = Qk(A)Pk(A)ro, Qk being a given polynomial . Inparticular, the case of the Bi-CGSTAB algorithm is treated in detail . Some otherchoices of the polynomials Qk are also studied .

AMS subject classification: 65F05, 65B05.

Key words : Linear equations, iterative methods .

1 Introduction.

We consider a system of n linear equations in n unknowns

Ax = b .

Let x0 be an arbitrary vector . We set ro = b - Axo and we define the numbersc2 by

ci = (y, Azro),

i = 0,1 . . . .

where y is an arbitrary nonzero vector and the linear functional c on the vectorspace of polynomials by

cW) = ci, i=0,1, . . .

Let Pk be the polynomial (assumed to exist) of degree at most k, such that

Pk(0) = 1

*Received April 1994. Revised October 1994 .

1 70

C. BREZINSKI AND M. REDIVO-ZAGLIA

andc(~'Pk (~)) = 0,

i = 0, . . ., k - 1 .

The family {Pk} is called the family of (formal) orthogonal polynomials withrespect to c [5] .The Lanczos method [35, 36] for solving Ax = b consists in constructing the

sequence of vectors rk = b - Axk defined by

rk = Pk(A)ro .

Since Pk(0) = 1, we can write

Pk(~) = 1 + eSk-1(~)

and thusXk = xo - Sk-1(A)ro .

As we shall see below, the vectors rk and Xk are usually computed recursively .An important property of the Lanczos method is its finite termination (if no

breakdown occurs), that is 3k < n such that rk = 0 and xk = x . Thus theLanczos method can also be considered as a direct method for solving a systemof linear equations .The polynomials Pk can be recursively computed by various relations, thus

giving rise to the methods known under the names of Lanczos/Orthores, Lanc-zos/Orthomin, Lanczos/Orthodir, biores, biodir, biomin or biconjugate gradientand so on, [22, 27, 32, 43] . See [16] for an extensive bibliography and for a unifiedderivation of all these algorithms based on formal orthogonal polynomials andsome new algorithms as well. The connection of these algorithms with Pade ap-proximants and continued fractions is treated in [27] . See also [28] . An accountof the history of the subject can be found in [26] .The conjugate gradient squared (CGS) method was proposed by Sonneveld

[41] . It consists in taking

rk = Pk (A) ro

and then computing xk such that rk = b - Axk .The Bi-CGSTAB algorithm of van der Vorst [42] consists in taking

rk = Qk(A)Pk(A)ro

with Qk (e) = (1 - ak-1~)Qk-1(~),Qo(O = 1 and choosing ak_1 such that

(rk,rk) = IIrk1I2 is minimum. Then xk is obtained as before .In this paper we shall consider some new methods of the same type where the

residual vector rk is defined by

rk = Qk (A)Pk (A)ro,

Qk being an arbitrary polynomial (not necessarily of degree k) such that Qk (0) _1 (this is a necessary condition for being able to compute xk from rk withoutusing A-1 ) .

Such a method will be called Biconjugate Gradient Multiplied, in short BiCGM .Obviously this class of methods includes the CGS and the Bi-CGSTAB as par-ticular cases. All these methods have the same property of finite terminationas Lanczos' if no breakdown occurs . Such methods were first considered al-most simultaneously in [6] and [29] . They are also called Lanczos-type productmethods .

We shall first propose some choices for the polynomials Qk. Of course, theonly choices of practical interest are those where either the product PkQk canbe computed recursively without computing separately Pk and Qk or where thisproduct is easy to obtain and does not require the storage of too many auxiliaryvectors . We shall consider the following methods

BiCG-Min

which uses least squares orthogonal polynomials

BiCG-Adj

which uses the adjacent family {pk l) }BiCG-Sti

which uses a weak form of Stieltjes polynomialsBiCG-SD

which combines conjugate gradient and steepest descentBiCG-Ort

which uses a second family of orthogonal polynomialsBiCG-e

which is based on the e-algorithmBiCG-S

which is Sonneveld's CGS methodBi-CGSTAB which is van der Vorst's method .

Then we shall show how to implement some of these methods of BiCGM typeand how to avoid breakdowns and near-breakdowns in the corresponding al-gorithms when the polynomials Qk are computed by a three-term recurrencerelation of the most general form .

2 The BiCGM class .

We shall now propose some possible choices for the polynomials Qk that seemof interest. Of course, other choices can also be studied . These polynomialsmust satisfy the condition Qk (0) = 1 in order that xk can be computed withoutknowing A-1 . Indeed, if this condition is satisfied we can write

Pk(e)Qk(e) = 1 + Uk(~) .

Thusrk = b - Axk = Pk(A)Qk(A)ro = (I + ATk(A))(b - Axo)

and we havexk = xo - Tk (A) ro .

These methods can be implemented either by some particular recurrence re-lations, as for the BiCG-S, or by implementing first the method of Lanczos,that is computing recursively the vectors rk = Pk(A)ro, and then taking a linearcombination of r'k , Ark, . . . with the coefficients of Qk . Several numerical testshave already been performed with the methods described below . They showthat these methods can exhibit, on some numerical examples, interesting con-vergence properties and are competitive with well established procedures . This

LOOK-AHEAD IN BI-CGSTAB

171

172

is the reason why we shall now present them even if they are not yet supportedby theoretical results and even if the details of their implementation have notbeen completely worked out . Obviously, their theoretical and practical studyhas to be pursued .

2.1 BiCG-Min .

Let Qkm) be the polynomial of degree at most k such that

'~

2~, [c(e Qk-) )~ = min,

m > k - 1 .i=O

Such a polynomial, called a least squares orthogonal polynomial, was intro-duced by Brezinski and Matos [7] .

We shall consider the polynomial Q( k) normalized by the condition Q(k) (0) _1 . Writing it as

Q( k) (~) = 1 - ak~

we have

ak = (co ol + . . . + CkCk+l)/(cl + . . . + ck+1) •

Setting

we have ak = /3k/ak

C. BREZINSKI AND M . REDIVO-ZAGLIA

ak

A

and

_ ~2 + . . . + Ck+ l,COC1 + "' + ckck+1,

ak+l = ak + Ck+2

/3k+1 = Nk+Ck+lCk+2, k=-1,0,1, . . .

with a_ 1 = X- 1 = 0 .The computation of Pk requires the knowledge of c0, . . . , c2k_1 . Far the com-

putation of a2k_2 and /32k_2 we need the same moments . Thus we shall take

(2k-2) (A)Pk(A)ro,

k = 1,2 . . . .rk = Q1

This method will be called BiCG-Min because of the minimization property ofthe least squares orthogonal polynomials Q( k) . Since no simple recurrence rela-

tion exists for the polynomials QkQ (1n) forarbitrary mand k, we restrictourselvours

the use of Q(k) which can be directly computed .


173

2.2 BiCG-Adj .

In this method, we shall make use of two adjacent families of orthogonal poly-nomials (which gives the name to the method) and take

Qk(e) = Pk l) ( ) _ kPkl) (S -1)

where {Pk l) } are the monic orthogonal polynomials with respect to the linearfunctional c( 1 ) defined by

CMW) = cW+1) = ci+l

i = 0, 1 . . . .

Since PL(1) is taken to be manic, Qk(0) = 1 .These polynomials Qk satisfy recurrence relations which are easily deduced

from those of P(1 1) . For example, the relation

PMk+1(~) = (S + Bk+l)Pk1)(~) - Ck+1 Pk1)1(e)

givesQk+1(e) _ (1 + Bk+l~)Qk(e) - Ck+10Qk-1(S),

and the relation

PMk+1(S) = (~+Ak+1)Pk1)(S) - Dk+lPk(~)

Qk+1(~) = (1 + Ak+1~)Qk(e) - Dk+1~Pk(S)

where Pk(C) = ~ k Pk(~-1 ) .

2.3 BiCG-Sti .

The Stieltjes polynomial Qk+1 of degree k + 1 is defined by

c(VPkQk+l ) = 0 for i = 0, . . . , k .

Usually such polynomials do not satisfy a recurrence relationship and theymust be computed directly by solving a triangular system of linear equations[39] . This is the reason why we shall restrict ourselves to polynomials satisfyingthe above condition only for i = 0, that is so that

C(PkQk+1) = 0 .

Of course, these polynomials are not exactly those defined by Stieltjes . It is easyto check that they satisfy a recurrence relation of the form

Qk+1(~) = (1 - ak~)Qk(~) , Qo(~) = 1 .

Indeed, we have

C(PkQk+l) = C((1 - ake)PkQk) = C(PkQk) - akc(~PkQk) = 0 .

gives

1 74


That isak = c(PkQk)/c(VkQk) .

Since ci = (y, A'ro) we have

ak = (y,rk)/(y,Ark),

rk = Qk(A)Pk(A)ro .

We can likewise consider the polynomials Qk+1 such that

c(VPkQk+1) = 0 for i = 0, 1 .

It can be easily checked that such polynomials satisfy a recurrence relation ofthe form

Qk+l (S) - (1 - akee)Qk(~) - bkWk- l(~)

where rk is defined as before and sk = Qk_ l (A)Pk(A)ro .

2.4 BiCG-SD .

Let us assume that the matrix A is symmetric positive definite . As explainedin [24], a projection method for solving Ax = b consists in the iterations

Xk+1 = Xk + akuk,

where Uk is an arbitrary vector and where ak is chosen such that (rk+l, Uk) = 0 .Since

rk+1 = b - Axk+ 1 = b - Ask - akAuk = rk - akAuk,

we haveak = (rk, uk)/(uk, Auk) .

The steepest descent (SD) method consists in choosing Uk = rk and thus

rk+1 = (I - akA)rk,

ak = (rk, rk)/(rk, Ark) .

Thus, if we set in this projection method rk = Qk(A)ro , then the polynomialsQk satisfy the recurrence relation

Qk+1(~) = (1 - ak~)Qk(e)

In analogy we can define a BiCGM method such that

rk = Qk(A)Pk(A)ro

where Qk is chosen such that (rk+l , rk) = 0. We obtain

rk+1 = (I - akA)Qk(A)Pk+l(A)r o .

with Qo(~) = 1 .

with Qo(~) = 1, Q_1O = 0 and

ak

(y, Ark)( ) - (

(y, Ask)

1) (

(y, rk)bk

(y, A2rk) (y, A2Sk) (y, Ark)


175

But Pk+1 = Pk - Ak~Pkl' with Ak = C(UkPk)`c(~UkPk l) ) where Uk is an arbi-trary polynomial of exact degree k and thus

rk+1 = (I - akA)Qk(A) [Pk (A) - AkAPkli(A)] r o .

If we set zk = Qk(A)Pk l)(A)ro then

rk+1 = (I - akA)(rk - AkAzk) .

Setting rk = rk - )kAzk we have rk+1 = rk - akArk and

(rk+1,rk) = (rk,rk) - ak(Ark,rk),

which gives

ak = (rk,rk)/(Ark,rk) .This method can be applied even if A is not symmetric positive definite .

This method must be compared with Bi-CGSTAB, where ak is chosen so that(rk+1, rk+1) is minimum [42], and also with the conjugate gradient of Hestenesand Stiefel [31] .

If A is not symmetric positive definite then AT A is non-negative definite andwe can take

rk+1 = (I - akATA)Qk(A)Pk+l(A)ro,

which gives, with the same notations as above

rk+1 = rk - akATArk,

ak = (rk, rk) / (Ark, Ark) .

This method must also be compared with Bi-CGSTAB [42] .

REMARK 2.1 . The iterates computed by the SD method (or its generalizationif A is arbitrary) have the form xk+1 = xk + akrk which shows that they fallwithin the framework defined by Mann [37] . Thus, as proved by Germain-Bonne [25], if Vk, 0 < ak < 1, then a necessary and sufficient condition that (xk)converges to x is that

k ~m ak = 0 and

ak diverges .k=0

2.5 BiCG-Ort .

Let d be a second linear functional on the space of polynomials . We can takefor {Qk} the polynomials orthogonal with respect to d, normalized by the usualcondition Qk(0) = 1 .

Of course, if d is identical to c, this method reduces to the CGS . A particularcase is to take for Qk the Chebyshev orthogonal polynomials which are known topossess acceleration properties, see [40] for a review . This method was alreadymentioned in [29] .

176


2.6 BiCG-e .Let us now consider the choice

Qk(~) = (1 - e)k = (1 - ~) Qk-1(~)

As explained in [16], this method corresponds to the second generalizationof Shanks transformation proposed in [4] . It can be implemented either bythe second topological e-algorithm (and thus the name of the method) or by anew algorithm generalizing the G-transformation [15] . These algorithms can beextended to systems of nonlinear equations .

2.7 BiCG-S .For the choice

Qk(~) = Pk(6)

we recover the CGS of Sonneveld [41] and we shall not discuss it further here .The problem of breakdown in this method was solved in [14] and that of near-breakdown in [9] .

2.8 Bi-CGSTAB.This method is due to van der Vorst [42] . It consists in taking

rk = Qk(A)Pk(A)ro

with Qk(~) = (1 - ak-1~)Qk_1(~) and Qo(~) = 1. We have

Pk+1 = Pk - Ak~Pk1 '

andQk+lPk+1 = (1 - ake)QkPk - \k~(1 - ak~)QkPk1) .

We set zk = Qk(A)Pk1) (A)ro . Then

rk+1 = (I - akA)(rk - AkAzk)

kak is chosen such that (rk+l , rk+1) is minimum, that is, if we set

rk = rk - A k Azk ,

f (ak) _ (rk+1, rk+1) = (rk - akArk, rk - akArk)

(rk, rk) - 2ak(rk, Ark) + a2(Ark, Ark) .

Thus df /dak = 0 if and only if ak = (rk, Ark)/(Ark, Ark) . Because of thisminimiza ion property, we have

(rk+l,rk+1) <- (rk,rk)

with Qo(~) = 1 .

and the Bi-CGSTAB appears as a particular case of the hybrid procedures in-troduced in [8] . This method must be compared with BiCG-SD, where ak issuch that (rk+l , rk) = 0 . A possible variant of this method is to take for Qk apolynomial of degree 1

Qk(S) = 1 - ak~,

to set rk = (I - akA)Pk(A)ro , rk = Pk(A)ro and to choose ak in order tominimize (rk, rk) that is

ak = (rk, Ark)/(Ark, Ark)

as before .

3 Implementation of the BiCGM .

3.1 The algorithm .

In the Lanczos method, the polynomials Pk are recursively computed. Thereare several possibilities for this computation, thus leading to the various algo-rithms which are found in the literature .

Let us first assume that all the polynomials Pk (as defined in section 1) existand are uniquely determined. A necessary and sufficient condition for that is Vk

HM


177

C1

C2

CkC2

C3

Ck+1=0 .

Ck Ck+1 . . . C2k-1

In this case the family of polynomials {Pk} can be computed by its usualthree-term recurrence relationship. Another possibility is to compute simulta-

neously the families {Pk} and { wherethe polynomials PM are the monic

orthogonal polynomials with respect to the linear functional c(') defined by

c(1) (~z ) = c(& l

) = ci+1,

i = 0, 1 . . . .

PM exists if and only if Hkl) 0 0 which is the same condition as for the existenceof Pk . We have [5]

Pk+l(~) = Pk (e) - Ak~Pkl) (O

with Po (e) = Pol)O = 1 and

Ak = C(UkPk)/c(eUkPk l) )

Uk being an arbitrary polynomial of degree k .

The polynomialsPkI)

can be computed either by their three-term recurrencerelationship (see section 2 .2) or by

PMk+1(~) = ( - L'k) Pkl) ( ) - Dk+lPk(e) .

There are also other possibilities which are reviewed in [16] .

1 78


If, for some value of k, the determinant Hk1) vanishes, then the correspondingorthogonal polynomial Pk does not exist and a division by zero occurs in oneof the recurrence relations used in the computation . This phenomenon is calleda breakdown, more precisely, a true breakdown [13] or a pivot breakdown [18] .A division by zero can also occur in one of the recurrence relationships evenif HJ 1) is different from zero, thus causing a so-called ghost breakdown [13] orLanczos breakdown [18] . It is possible to avoid breakdowns in the algorithmsfor implementing the Lanczos method (and similarly in the BiCGM) by jumpingover the non-existing polynomials and computing only the existing ones (calledregular) . This is the technique followed in [12] for avoiding breakdown in Lanczostype methods and in [14] for the CGS . Let us mention that, due to the different

normalization of the polynomials Pk1) which are assumed to be monic, Pk exists

if and only if Pk1) exists .Now, in the recurrence relations, if a quantity in a denominator is not exactly

zero but close to it, then a severe propagation of rounding errors can affectthe algorithm, a phenomenon called near-breakdown . In order to avoid a near-breakdown, one can jump over the badly computed polynomials and computedirectly the first regular polynomial following them . This was the procedurefollowed in [10, 11] for the Lanczos method and in [9] for the CGS . Let usmention that other look-ahead procedures for treating these problems exist inthe literature [1, 18, 2, 23, 28, 30, 34, 38] . However, since these procedures werederived by purely linear algebra techniques, we wonder if they could be easilyadapted to the algorithms of the BiCGM class. As we shall see now, our approachfollowing formal orthogonal polynomials, provides a solution to the problems ofbreakdown and near-breakdown in BiCGM methods .

Let us first give the recurrence relations used in that case. For that, we shallchange our notations and give a name only to the polynomials which are used

in the recurrence relations . Thus, now, we shall denote by PV ) (instead of P(,1~) ~the monic polynomial of degree nk > k belonging to the family of orthogonalpolynomials with respect to c( 1 ) . Pk (instead of Pnk ) will denote the polynomialof degree nk at most belonging to the family of orthogonal polynomials withrespect to c. Since similar jumps in the degrees will be considered for the aux-iliary polynomials involved in the methods, we shall also denote by Qk (insteadof Qnk ) the corresponding polynomial used in the BiCGM-type method .

Thus, at each step k, we first have to decide the length Mk > 1 of the jump .The tests for deciding when and how far to jump will be discussed in detail in thenext section . We set nk+1 = nk + Mk . Then, the following recurrence relationshold [10]

(3 .1)

(3 .2)

Pk+l(~)

evk(~))Pk(~) - ~wk(C)Pk1)(~),P(l)k+1(~) = qk (~)Pkl) (6)+ tk (6)Pk (~),

where wk is a polynomial of degree Mk - 1 at most, vk a polynomial of degree

Mk - 2 at most if nk - Mk + 1 > 0 and nk - 1 otherwise, qk a monic polynomialof degree Mk and tk a polynomial of degree Mk - 1 at most if nk - Mk > 0 and


179

nk - 1 otherwise. In the case of breakdown, vk and tk are identically zero . Thecoefficients of these four polynomials are obtained as the solutions of systemsof linear equations, see [10, 11] . In order that these relations hold, it must beassumed that c(~n'kPk) zA 0 .In order to show how to avoid breakdown and near-breakdown in BiCGM

methods, we shall now assume that the polynomials Qk satisfy a recurrencerelationship of the form

(3.3)

Qk+1(O = Ek(~)Qk(~) +eFk(~)Qk-l(~)

with Q_ 1 (~) = 0 and Qo(e) = 1. Ek and Fk are arbitrary polynomials ofarbitrary degrees. Since we need to have Qk+1 (0) = 1, we must have Ek(0) = 1 .In many of the methods suggested in section 2, the polynomials Qk satisfy

a recurrence of this type . We shall now see how the product QkPk can berecursively obtained .Multiplying (3 .1) by (3.3) gives (dropping the variable ~ when unnecessary)

Qk+lPk+1 = Ek (1 - evk)QkPk - ~EkWkQkPk l)

+ ~Fk(1 - SVk)Qk-1Pk - e2FkwkQk-1Pk l) .

Thus, we also need to compute recursively the products QkPkl) , Qk-lPk and

Qk_1Pk 1) . Multiplying (3 .2) by (3.3), we have

(3.5)

Qk+1Pk+1 = EkgkQkPkl)+ EktkQkPk + SFkgkQk-1P( 1)

+ VkFkQk-1Pk .

Multiplying (3 .1) and (3.2) by Qk leads to

(3 .6)

QkPk+1 = (1 - ~Vk)QkPk - SWkQkP,

QkPk+1 - gkQkPk l) + tkQkPk .

rk = Pk(A)Qk(A)ro,

Zk = Qk(A)Pkl) (A)ro

sk = Qk-1(A)Pk(A)ro,

Pk = Qk-1(A)Pk l) (A)ro

(3.4)

(3 .7)

Thus if we set

the relations (3.4), (3 .5), (3.6) and (3 .7) give immediately

(3.8) rk+1 Ek(A)(I - Avk(A))rk - AEk(A)wk(A)zk

+ AFk(A)(I - Avk(A))sk - A2Fk(A)wk(A)pk(3.9) zk+1 Ek(A)gk(A)zk + Ek(A)tk(A)rk + AFk(A)gk(A)pk

+ Atk(A)Fk(A)sk

(3.10) Sk+l = (I - Avk (A))rk - Awk(A)zk

(3.11) Pk+1 = gk(A)zk + tk(A)rk .

1 80


Let us now see how to compute xk+1 from rk+l . Since Ek(0) = 1, we canwrite

Ek (~) = 1 + ~Rk (e)

Thus

Ek(1 - ~vk) - (1 + ~Rk)(1 - ~vk) - 1 + e(Rk - vk - SRkvk)

and it follows, since ~Rk = Ek - 1

xk+l = xk - (Rk(A) - Ek(A)vk(A))rk +Ek(A)wk(A)zk

-Fk(A)(I - Avk(A))sk + AFk(A)wk(A)pk .

Let us mention that, in (3 .3), the variable 6 is needed in the second term ofthe right hand side in order to be able to compute xk+1 from rk+1 without usingA-1 .In the case of CGS, the formulae given in [9] are recovered . Thus we are able

to avoid breakdown and near-breakdown in all the methods of the BiCGM-typeand, in particular, in the Bi-CGSTAB method of van der Vorst [42] which seemsto be very efficient and promising .At each step, the number of matrix-by-vector and inner products and the

storage requirements depend on the degree of the polynomial Qk. For example,in the case of CGS, we need 6mk - 3 matrix-by-vector products if Mk < nk + 1,and 5mk + nk - 2 products otherwise and the storage overhead is 0 (mkn) .

3.2 Avoiding AT .

One of the main interest of CGS [41] and Bi-CGSTAB [42] is that their imple-mentation avoids the use of AT . In [14] and [9], it was respectively shown how toavoid the breakdown and the near-breakdown in the CGS without using AT. Weshall now explain how to avoid the use of A T when curing breakdown and near-breakdown in the algorithms of the BiCGM class in the particular case wherethe polynomials Fk are identically zero which is a case covering Bi-CGSTAB . Weshall not give here all the details since they are quite similar to those given in[9] where they can be found .The coefficients of the linear systems giving the coefficients of the polynomials

vk, wk, qk and tk involved in (3 .1) and (3.2) have the form

c(C k +ZPk ) and c(1) ( 1k+ip(1)'

for i = 0, . . . , 2mk - 1 . We have, from the definition of the moments of the linearfunctional c given in section 1,

c ( tZZk+iPk) _ (y,A1k+i Pk(A)ro), c(l)(~'hk+ZPkl)) _ (y,Ahk+Z+

1 Pkl) (A)ro ) .

In order to avoid the computation of quantities like (y, Ank+jz) for high valueof nk + j, such scalar products could be rewritten as (AT'k y, Ajz), but such

expressions imply using AT . We shall now see how to avoid this drawback, evenif a relation such as (3.3) does not occur among the polynomials Qk .Let us assume now that Vk, Qk has exactly the degree nk . We set

Qk (~) = 1+a (k)~ + . . . .+ a (k) ~nk

and recall that (3 .1) holds, that is

(3.12)

where wk is a polynomial of degree Mk -1 at most and vk a polynomial of degree

Mk - 2 at most (if Mk G nk) and of degree nk - 1 at most (if Mk > nk) .Let us now explain how to obtain the coefficients of the polynomials vk and

wk without having to use AT . Two cases have to be considered according to therespective values of nk and Mk .

•

Case nk - mk + 1 >0--We set

Wk (~) = TO + . . . + .ymk _ 1~mk-1

There exists a polynomial U1 of degree nk - 1 and a constant /3'k-1 such that

~nk-mk+lWk(~) = Ul(e) +Omk-1Qk(~) .

Identifying the coefficients of ink in both sides of this relation, we obtain

k7mk-1 = ~mk-1 of k

There exists a polynomial U2 of degree nk - 1 and two constants ,3,,,_j and/3m, k -2 such that

vk-rnk+2Wk(~) = U2(0 + )mk-2Qk(e) + ~mk-1SQk( ) .

Identifying the coefficients of ~nk leads to

(k)

( k )^ymk-2 - On,-tan k + 0mk-lan k -1

and identifying the coefficients of ~nk+1 we have the same relation as above for-ymk _ l , which proves that I3m, k _ 1 is the same as above .And so on, there exists a polynomial Umk of degree nk - 1 and constants

/3o, . . . , /3,,, - 1 such that

Vzwk(S) = Umk(~) +(3oQk(~)+ . . . +)3mk-lemk-lQk(e)-

Identifying, we obtain one more new relation

'yo = /3oa(k) + 01a(k)-1 + . . . + 0.,-1a(k)-mk +1


Pk+l(e) = Pk(~) - b Wk(~) Pk l) (~) - ~ Vk(e) Pk(~)

= (1- evk(ee)) PkO - ~'wkOP( 1) ( )

181

182

with a0

(k) = 1 .-We set

vk (c) = yo + . . . + •Ymk-2Smk-2 .

Now, similarly, there exists a polynomial V1 of degree nk - 1 and a constant'/3mk-2 such that

e'2k-mk+2vk( ) V1(0 +/mk-2Qk( )'

By identification, we have

_ ~~

a(k)ymk- 2

mk-2 nk .

There exists a polynomial V2 of degree nk - 1 and constants O.k-2 and13.1 k-3

such that

S nk mk+3 vk(b) = V2(0 +'-mk-3Qk(~) + 0mk-2Wk( ) •

Identifying, we have

_ ~

a(k) + ~/

a(k)Fmk -3 = ~mk-3 nk

mk - 2 nk-1'

And so on, there exists a polynomial V,,, k -1 of degree nk - 1 and constants010 ,

0~

'

such that. . . , ~3712k -2

bnk vk(S) = Vmk -1(~) + IOQk(~) + . . . + 0mk-2S mk 2Qk( )'

By identification, we obtain

yo = ~Q a ( k) + ~ia(k) I + . . .+ 0111k-2a{k)-mk+2 .

The preceding relations allow us to compute the y's and the y"s from the ~3'sand the /3"s. Let us now show how to compute the /3's and the ,Q"s .

We apply the orthogonality conditions of Pk+1 with respect to S' for i = nk -mk+l, . . . , nk-l . Writing down the orthogonality condition c(~nk-mk+lpk+1) _0, we obtain


C (~nk-mk+1Pk+1 ) C (tnk-mk+lpk) - C(1) (~nk-mk+lwk p( 1))

-C (enk-mk+2vkpk) = 0 .

Expressing wkPk 1) and VkPk in terms of the U,'s and the V's by means of the

relations given above and using the orthogonality of Pk and P(1), we get

C(1) (Ulpkl)) )+O""-1C(1) (Qk PP') + C (V1Pk) + 13',l k-2 C (Qk Pk) = 0 .

But, since U1 and Vl have degree nk - 1, then c( 1 ) (U1Pk1) ) = c(V1 Pk) = 0,

and it follows


183

Nmk-1C (1) (QkPk l)) + 0mk-2 C (Qk Pk) = 0 .

Writing down the orthogonality condition c (enk-mk+2Pk+1) = 0 and using

c( 1 ) (U2Pk 1)) = c (V2Pk) = 0, we obtain

/3mk-2C(1) (QkPk l) ) + /3mk-lC(1) (~QkPk 1)) + 0mk-3C (QkPk)

+/3m,-2C (MPk) = 0 .

And so on until c(~'k-IPk+1) = 0 which gives

01 CM (QkPk 1) ) + . . . .+ Olnk-ic (l) (ck-2QkPk l) ) + / O'C (QkPk) + . . .

+0,n,-2C (Lmk 2QkPk) = 0 .

Thus we have obtained Mk - 1 relations for determining the 2mk - 1 unknowncoefficients 3i and /3i' .We shall now write the remaining orthogonality conditions of Pk+1 as

c V nkQkPk+1) = 0

for i = nk, . . . , nk +mk - 1 .

Thus we obtain for i = 0, . . . , Mk - 1

C (1) (b2wkQkPk1)) + c (V+1vkQkPk) = c (VQkPk) ,

that is

'Yoc(1) ( ZQkp(1)) + . . .+ ymk _1c(1) (~i+mk_1Qkp(1)) +

yoc ( a+1QkPk) + . . . + Fmk-2C (Stz+mk-lQkPk) = C (CQkPk) .

Replacing, in these relations, the y's and the y"s by their expressions above, weobtain mk additional relations for the /3's and the /3"s . Setting

ci = c(1) (NkP (1) )

di = c (VQkPk)

these Mk relations become (for i = 0, . . . , mk - 1)

'YOCi + . . . + ymk-1Ci+mnk-1 + 1Odi+1 + . . . + y,,i k 2di+mk-1 = di,

that is

q (3oa(k).+ . . . + 3mk_lank)-mk+1) + . . . + Ci+mk-1 (/3m k -la n k) ) +

di+1 (C3oa ( k) + . . . + 33Mk-tank)-mk+2) + . . . ++ di+mk-1 ( mk-2 ank)) = di,

184


or

/3o (cia (k) ) + 01 (cia (k)-1 + ci+1a (k) ) + . . .

+0'.'-1 (ciank)-mk+1 +. . . + Ci+mk-1a

(Thk)) +

~o (di+la (k) ) + 0 , (di+la(k)_1 + di+2a(k)) + . . .

+ Nmk -2 (di+tan k)_mk+2 + . . . + di+mk-la 7Lk)) = d .

Thus, we have the same number of unknowns and relations .

Case nk-mk+1<0.We have again

wk (O _ 'Yo + . . . + "/Mk- 1~'mk-l

Since nk < Mk - 1 and wk is a polynomial of degree at most Mk - 1, thenthere exists a polynomial U 1 of degree nk -1 and constants hank, . . . , ~3,,,, k -1 suchthat

Wk(S) = U1(~) + OnkQk(e) + . . . + Omk -1~mk-nk-I Qk(S) .

Identifying the coefficients of the successive terms in S' for

we obtain (with the convention that at~k) = 0 for Z* < 0)

(k)

(k)

(k)Ink = )3nkank + Onk+lan k -1 + . . . +- / 3mk - 1 a2nk-mk+l,

(k)

q

(k)

(~

(k)Ink+1 = ~nk+lank + h'nk+tank-1 + . . . + Nmk - la2nk -mk+2 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

'Ymk-1 = Nmk-lan k) •

There exists a polynomial U2 of degree nk - 1 and constants j3n k-1, Nnk, . . .Nmk_1 such that

~wk(~) = U2(~) + /3nk-1Qk(~) + . . . + 0mk-1~mk nk Qk(S)

By identification we obtain

/~

(k)

(~

(k)

(~

(k)ynk - 1 = Nnk-lank + ~"nkank-1 +' . . +. Nmk -la2nk-mk

and the same relations as above for Ink , . . . , ymk -1 .And so on, there exists a polynomial Unk+1 of degree nk - 1 and constants,3o,...,/3,,,,-1

such that

C`wk(S) = Unk+1(S) +OoQk(~) + '+ pmk-1enzk-1 Qk(0-

Identifying, we obtain

'Yo = 0oank) + O1a(k-1 + . . . + 0,,,,-1a (k)-mk+1

)

i = nk, . . . )Mk - 1,

LOOK-AHEAD IN BI-CGSTAB 185

-In this case we shall take for vk a polynomial of degree at most nk - 1 (fork = 0, n o = 0 and there is no polynomial vo) and we set

Vk(S)=y0+ . . .+yn

nk-1

Similarly, there exists a polynomial V1 of degree nk - 1 and a constant 0.1 k -1such that

~vk(S) = V1(~) +Onk-1Qk(~) .Identifying, we get

r

(k)ynk-1 - lnk_1 ank'And so on, there exists constants 010 ,

and. a polynomial Vnk of degreenk - 1 such that

S nkvk(~)

V"nk(')+Oo'Qk(~)+ . . .+Onk-lSnk-1Qk(~),

which gives, by identification

'yo = /boa(k) + ~la (~)-l + . . . + ,3. k - lal k) .

Writing down the orthogonality conditions c (eiPk+1) = 0 for i = 0, . . . , nk - 1,and using c( 1 ) (U1 Pk' )) = c(V1 Pk) = 0, we obtain

On,c(1) (QkPkl)) +. . . + 0'~nk - 1 C(1) (e mk-nk-1QkPk1)) +

1"nk-l C (Qk Pk) 0 ,

Onk-1C (1) (QkPkl)) + . . .+,~3,nk-1C(1) (S mk-nk QkPkl)) +

~-'nk-2C (QkPk) + Nnk-1C (WkPk) = 0,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

01c(1) (QkPk l) ) + . . . + ~3mk-1c(1) ( " k-zQkPk1) ) + Ooc(QkPk) + . . . +

Ok-lC(~nk-lQkPk) = 0-n

Thus, we have already obtained nk relations .Now, writing down the conditions

C (VQkPk+1) = 0

for i = 0, . . . , mk - 1, we obtain

Yoc (1) (NkPkl) ) + . . . + ymk-lC(1) (ei+mk_1QkPk1)) +

y0c (ti+1QkPk) + . . . + _y.,-1C(~z+nk QkPk) = C ( ZQkPk) .

Replacing, as above, the yi and the yi by their expressions, and using the ci'sand the di's previously defined we obtain the following Ink additional relationsfor the Oi and the ,QZ

/̂OCi + . . . + 7mk-1Ci+ink-1 + y0di+l + . . . + ynk-ldi+nk = di,

186


that is

or

Ci (3oa((k+ . . . ~jmk -lank)-mk+l) +. . . + Ci+mk-1 (Omk-lan k) ) +

di+1 (~ca(k)

. . . + rck lalk)) + . . . + di+nk (ink-lank)) = di,

ia(k)) + 01 (cia (k)_ 1 + ci+ia(k)) + - - • +

~mk 1 (cia (k)_nek+l . . . + Ci+m -la( ~)) + Qo (di+la(k)) +

~i (di+ta(nk)-1 + di+2a (~)) + . . .

+' nk-1 (di+ialk) + . . . + di+nka (k)) = di .

Thus, finally, we have nk + Mk relations with the same number of unknowns .

Let us now try to compute recursively Pk+ 1 . We shall look for Pk+1 under theform

(3.13)

Pk+1(~) = qk(~) Pkl) (~) + tk(~) Pk(~)

where qk is a monic polynomial of degree Mk and tk a polynomial of degreeMk - 1 at most (if Mk < nk) or of degree nk - 1 at most (if Mk > nk) . Forcomputing the coefficients of the polynomials qk and tk two cases have to beconsidered according to the respective values of nk and Mk .

•

Case nk - Mk > 0 .We set

qk (~) = no + . . . + qmk-1 tmk-1 + ~mk .

There exists a polynomial Ul of degree nk - 1 and a constant amk such that

tnk-mk qk (~) = Ul (e) + am, Qk(e)-

Identifying the coefficients of enk in both sides of this relation, we obtain

1 = a a(k)mk nk

There exists a polynomial U2 of degree nk-1 and two constants am, and am,_1such that

Snk-'mk+lqk(~) = U2(~) + amk-1Qk(~) + amkWk(S) •

Identifying the coefficients of ink leads to

nmk-1 = am k-la(k)nk + am ka(k)nk _1

and identifying the coefficients of ink+l we have the same relation as above whichproves that amk is the same as above. And so on, there exists a polynomialUm, +j of degree nk - 1 and constants ao, . . . , CY,nk such that

S nk gk(b) = Umk+l(~) + aoQk( ) + . . . + am,emk Qk( ) .

Identifying, we obtain one more new relation

no = aoa (k) + ala (k) 1 + . . . + amkank)_mk .

We set

tk(c) = 7710 + . . . + nmk_1c'nk -1

Now, similarly, there exists a polynomial Vl of degree nk - 1 and a constantam k _ 1 such that

Cnk-mk+ltk(0 = V1(0 + amk-1Qk(~) .

By identification, we have

_ i

(k)77,nk-1 - amk_lank .

And so on, there exists a polynomial Vmk of degree nk - 1 and constantsao, . . . , a mk_ 1 such that

~nktk(e) Vm-k (~) + aoQk(e) + . . . + amk-1S-k-1

Qk(~)

By identification, we obtain

77o = aoa( k) + 0 1 a (k)_ 1 + . . . + aMk - l ank)-,nk+l .

The preceding relations allow us to compute the rd's and the 77"s from the a'sand the a"s . Let us now show how to compute the a's and the a"s .We apply the orthogonality conditions of Pk+1 with respect to ~i for i = nk -

Mk, . . ., nk - 1. Writing down the orthogonality condition c(l) (~nk-'nkPk+1) _0, we obtain

c(1) (~n'k-mkPk1F)1) = C(1) (~nk-mkgkPk1)) + c (~nk-mk+ltkPk) = 0 .

Expressing gkPk 1) and tkPk in terms of the UZ's and the V's by means of therelations given above and using the orthogonality of Pk and P(1), we get

c (1)

l)(UiPk) + amk c (1) (Qk PV ) ) + c (V1Pk) + am k -lc (Qk Pk) = 0 .

But, since U1 and V1 have degree nk - 1, then c( l ) (U1Pk 1) ) = C (V1 Pk) = 0,and it follows

amk-lc (Qk Pk) = -a, kc(1) (QkP(k1)) .


187

1 88


Writing down the orthogonality condition c(1) (~nk-mk+lPk+l) - 0 and using

c(1)(U2Pk1)) =c

(V2Pk)= 0, we obtain

am,-lc(1) (QkPk 1) )+a'„Zk -2C (QkPk)+am k -lc (eQkPk) = -amkC(1) (QkPP1) ) ,

and so on until c (1) ( {nk-1 Pk+1) = 0, which gives

alc (1) (QkPkl) ) + . . . + a k -lc(1) (Ch-2 Qkp(1) ) + apc (QkPk) + . .

+at

c ( mk-1Qkpk) =-amkc (1) (tmk -1Qkpk 1 ))

mk-1

Thus we have obtained Mk -1 relations for determining the unknown coefficientsai and ai .

We shall now write the remaining orthogonality conditions of P( 1) as

C (1) (NkPk+1) = 0 for i=0,.-.,Mk-1.

Thus we obtain

C (1) (e igkQkPk1) ) + c (&1tkQkPk) = 0

that is

noc (1) ( iQkPk1)) + . . . .+,7mk-iC(l) (~i+mk-1Qkpk1)) +

770C (~2+1QkPk) +. . . + 77'k_1C

(S 2+mk Qkpk) = -C (1) (SZ+mkQkPkl) ) .

Replacing, in these relations, the 77's and the q"s by their expressions above, weobtain Mk additional relations for the a's and the a"s, that is

ci (aoa(k) + . . . + a n , an k)-mk ) + . . . + ci+mk-1 (,m,-lank) + a,,ak a, k _1)

+' di+1 (a pank) + . . . + ark-la(k)-mk+l) + . . . + di+mk (amk- 1 a(k))-Ci+nn k ,

or

ao (cia(k)) + a (c .a(k) + ci+ia(k)) + . . . + am 1 (cia (k)

+ . . .nk

1 2 nk-1

k

k-

n7, -mk+1

+ ci+m k-la(k)) +a0 (di+ia (k) ) + a, (dZ+ia(k)- 1 + di+2a(k)) + . . -

+ Q:mk-1 (d2+1a'nk-mk+l + . . . + di+mka(k)) -

- am,, (ciank_mk + . . . + ci+mka(k)) -

e have the same number of unknowns and relations .

• Case nk - Mk < 0-We have again

Rk (~) = 77o + . . . +,qmk-1Smk-1 + tmk

There exists ao, . . . , amk _1 such that the q2 are given

S

by

(k) +

( k )77mk - I = amk-lank 7 C Mk ank-I(k)

(k)

(k)1lmk-2 = a.,-2an k + 'Mk - lank_1 + amk ank - 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71o = aoank + alas k)-l + . . . + amk-la (k) m"+1 + amk ank)-m

Moreover, there exists a polynomial U1 of degree nk - 1 such that

4k(~) U1(~) + ankQk(~) + . . . + amk~mk-nk Qk(e)

Identifying the coefficients of Cmk in both sides of this relation, we obtain againb

1 = amk af k) •

-We shall take for tk a polynomial of degree at most nk - 1 (for k = 0, no = 0and there is no polynomial to)

tk(S)_~I+ . . .+71, k - 1z

nk -1 .

We have

(1)77nk-1 = ank-lank. . . . . . . . . . . . . . . . . . . . . . . .

77 o - aoa ( k) + al a~k)_ 1 + . . . +ank -lalk) .

Applying the orthogonality conditions we have

ankc(1) (Q k p(1)) + . . . + amkc(1) (Cmk nk QkPk 1) ) + a 'nk -lC(Qkpk) = 0,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

alc(1) (QkPk l) ) + . . . + amkc (1) (emk-1QkPkl)) +

aoc (QkPk) + . .. + a' k_lc (ink -lQk pk) = 0.

We have also

77oc(1) (eQk pkl)) + . . . + 17m'-lc(1) (ei+mk-1Qkpk 1) ) +

C(1) (SZ+nk Qkpk l)

)+ 1Jpc (SZ+lQkpk) + . .

. +ink-lC (&nkQkpk ) = 0

for i=0, . . .,mk-l .


189

190


Replacing, in these relations, the i7's and the 11"s by their expressions above,we obtain

ci (apa(k + . . . + a,,,a(k)_mk) + . . . ci+mk-l (am,-lank) + (kmkCL{ k)-1)

+ di+ 1 (a0 nk) + . . . + aflk-lalk) ) + . . . + di+nk (ank-lank)) = - Ci+mk ,

that is

as (cia(k)) + ai (cia{(k)_I + ci+l

+ . . .

+ a,,, k 1 (ciank)-mk+l + . . . + ci+mk-lank)) + a0 (di+lank)) +

al (di+la~k)-1 + di+2a (k)) +. . . + ask-1 (di+lal k) + . . . + di+nka( k

) )

-

(

k

(̀kk (CZank-mk + . . . + Ci+mk-lark-1 - C2+mk,

or

c 0 (cia (k) ) + OZ, (cia (k)-i + ci+lank)) + . . .

+ amk-1 (ciank)-mk+l + . . . + ci+'mk -lank) ) + a0 (di+lank) ) +

al (di+1a(k)-1 + di+2a(k)) + . . . + ank_1 (di+lalk) + . . . + di+,ikank) ) _

k- Cemk cian k-mk + . . . + Ci+mk a(k)

In what precedes, we assumed that the polynomial Qk has the exact degreenk. A quite similar treatment can be made if Qk has the degree qk < nk . Thiswill be the case, for example, in the line minimization described in section 4 forthe Bi-CGSTAB, where qk = k . In that case, writing

Qk(~) = 1 +a(k) e + . . . + a((k)~gk

we obtain, when nk - Mk + 1 > 0,

Fmk-1 = mk_1agk )

ymk-i - /3mk-iagk) + . . . + 01nk_laqk-i+l for i = 2,- ., ,Mk

with the convention that ai = 0 for i < 0 and

(~

a(k)Nmk - 2 qk(k) k

7m. k -i = ~3,nk - iagk + . . . + )3m,-lag,-i+2 for i=3, .

The ,di and the ,(3i are the solution of the system

,3,.,-ic.k-qk +' mk-2dnk-qk - 0,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

r131C-qk + . . . + N(~mk-lCnk-4k+mk-2 + 3Odn,-qk + . . . +

'"mk-2dnk - qk+mk- 2 = 0 ,T0c + . . . + ymk-1Ci+mk-1 + YOd2+1 + . . . + Fmk-2dz+mk-1 = di

i=nk - gk, . . .,nk - qk+mk - l,

7~mk_2 =


191

after replacing, in the last Mk relations, the -yi and the yi by their expressionsgiven above .

The case nk - Mk + 1 < 0 can be treated similarly and the recurrence rela-tionship (3 .13) as well .

However, it must be noticed that these relations involve the quantities

C( 'nk-qk+2QkPk),

C(I) (~nk-qk

k+2QkP(1) ) 7

i - 0, . . . . 2mk - 1,

which means that the vectors

An,-q,+iQk(A)Pk(A)ro,

Ank-qk+i+I Qk(A)P')(A)r,,

will have to be computed for i = 0, . . . , 2mk -1. Thus, if the difference nk - qk islarge the procedure will be costly or will necessitate the use of AT. In particular,if qk = k, as in the line minimization described in section 4, and if many jumps orlong ones occur, the discrepancy nk-k will be large and the procedure expensive .

3.3 Summary of the algorithm .

Let us now briefly summarize the computations to be performed in our al-gorithm. At the step k, after having decided the value of Mk (see the nextSubsection), we have to

•

Compute the O's and the Q"s,

•

Compute the y's (coefficients of wk) and the y"s (coefficients of vk),

•

Compute rk+1 by (3.8) and control its value for deciding to continue ornot,

•

If the solution has not been obtained

- Compute sk+ 1 by (3 .10),

- Compute the a's and the a"s,

3.4

Compute the it's (coefficients of qk) and the ?"s (coefficients of tk),

Compute zk+1 by (3.9) and Pk+1 by (3.11) .

Tests for near-breakdown .

Let us first consider the case where some of the Hankel determinants definedin section 3.1 vanish . It was proved by Draux [21] that

c (I) ~ iPk l) ) = 0 for i = 0, . . . , nk + mk - 2

and thatC(1)(~nk-I-+nk-IP('))

O .

1 92


These relations define the value of Mk in the case of breakdown . Thus, it seemsthat the near-breakdown could be defined by replacing 0 by some threshold ethat is

c (i) (~ iPki)) < e for i = 0, . . . , nk + Mk - 2

and

c(1)I Fnk+mk-1p(1)

)

However, as already observed

/

with the CGS [9], such a test is not satisfactoryand it must be replaced by a more suitable one . Following [9], let us set

(1)ak1 = c(1) (QkPk+l) .

It is easy to see that

a (1) =a(k)c(~nk}1Pk+1)=k+l

nk

Thus o,( ' ) l is zero (or close to it) in three different cases

1. when the solution is about to be obtained, since ak+l = 0 if Pk+1 (A)ro = 0 ;that is if Xk+1 = x,

k) =2. when of

0,

3. when C(~nk}1 Pk+1) = 0 .

From the last two cases, we see that ak+1 ~ 0 if and only if a~k) :,~ 0 andc(~nk+1Pk+1 ) :A 0, which are the two conditions for not having a breakdown (ora near-breakdown) at the next iteration of the algorithm . Thus, after testing

if the solution has been obtained, a jump has to begin if ak+i I < e. So the

value of Mk is set to 2 and the system giving the coefficients Oi, J3i, ai and a'is solved. If this system is singular (pivot= 0) or nearly singular (Ipivotl < e l )

then Mk is changed to Mk + 1 and the procedure is repeated until a non-nearlysingular system has been found. But, it is also necessary to check whether ornot a near-breakdown will occur at the beginning of the next step in order toavoid it (otherwise it will be too late) . For that, we set

ak+1)- C(~mk QkPk+1) .

We have

ak+1 ) = an k) c( nk+ 1Pk+1 ) .

Thus, if the solution has not been obtained, the length Mk of the jump hasstill to be increased in the current step, the system giving the coefficients of thepolynomials has again to be solved and the whole procedure has to be repeated

until all the conditions be satisfied . It must be remarked that ak+i) is computedby

ak+1 = ( y, A-- sk+l),

with sk+1 = Qk (A)Pk+1(A)ro .The other tests mentioned in [9] also have to be performed .Of course, the tests we used are only based on heuristics and there is no

guarantee that they will work well in all situations nor that the systems giving thecoefficients of the polynomials involved in the recurrence relations will be well-conditioned (although we test the pivots in the Gaussian elimination procedureused for their solution) . It may be possible that better tests could be based onthe results obtained by Beckermann [3] .

4 A look-ahead Bi-CGSTAB.

In the Bi-CGSTAB introduced by van der Vorst [42], the polynomial Fk isidentically zero and, when mk = 1, the polynomial Ek is written as

Ek(e) = 1 - bT;,

and the parameter b is chosen so that (rk+1, rk+1) is minimized that is

b = (sk+1, Ask+1)/(Ask+l, Ask+1),

Sk+1 = Qk(A)Pk+1(A)ro .

Let us now explain how to compute the coefficients of Ek when Ink > 1 .There are three different possibilities . The first one consists in taking for Ek apolynomial of degree one, even if Mk > 1, that is defining it as van der Vorst[42] . In the two other cases, Ek will be a polynomial of the degree Mk at most .

1 . Line minimization

Ek(t;) = 1 - beFor x0 = 0 and y = (0, 0, 0, -l, 1, . . . , -(-1)n-1 , 0) T, a breakdown occurs at

the first iteration when n is even since (y, ro) = (y, Ar o) = 0 .For n = 400, xo = 0, e = 10 -30 we have a jump of length 2 at the first iteration

to avoid the breakdown and we obtain n54 = 55 and 11r 54 [1 = 0.66 x 10-11 whichcoincides with the actual residual (that is the residual computed from xk byrk = b - Axk) . When e = 10 -20 and a2 = 10-14 , we have again the jump oflength 2 at the first iteration but we also have several jumps (from n34 = 35 ton35 = 37, from n35 = 37 to n36 = 40, from n36 = 40 to n 3 7 = 42 and, finally, ajump from n37 = 42 to n35 = 48) and we obtain 11r3811 = 0.42 x 10-14 and anactual residual of 0 .62 x 10 -14The behavior of the algorithm for n = 400 and these two different values of e

.s shown in Fig . 1,e 1 Az

sk+1, Ask+1) + " . + emk (A Zsk+1, A'nk sk+l) = -(AZ sk+l, Sk+1)

for i = 1, . . . , Mk, where sk+1 is defined as above .

3 . Step-by-step minimizationWe set


193

Ek(~)=(1-bmk~)' . . . •(1-b1~)=1+ei~+ . . .

1 94


We also set

rk+1 = sk+i,

rk+i = ( 1-biA),k+1), for i=1,---,Mk-

We choose bi which minimizes (rk+l , rk+1), that is

bi = (rk+1 ) Ark+1))~(Ark+1 ~, Ark+11 ),

and we obtain rk+1 = rk+i ) .A crucial point which must be noticed is that the vector sk+1 depends on the

value of Mk . Thus, when changing Mk, the new vector sk+i has to be computedbefore beginning the minimization process .

5 Numerical results .

In this section we shall present some numerical results obtained with our im-plementation of the look-ahead Bi-CGSTAB of section 4 . The program, writ-ten in Fortran 77, uses the theory explained in the previous sections and thebreakdowns and near-breakdowns are detected using the tests described in thesubsection 3 .3 and those already reported in [9] . The results were obtained ona SUN SparcStation IPC . Our results were compared with those obtained byimplementing the usual Bi-CGSTAB algorithm proposed by van der Vorst in[42] .As in [9], we use three different thresholds

•

E for testing the quantities involved in the jumps,

•

E1 for testing the pivots and the determinant in Gaussian elimination andjumping,

•

E2 for testing if JrkJJ < E2 and stopping .

The numerical results show that, in the case of the look-ahead Bi-CGSTAB,the choice of El is not an important item and that a very small value for thisthreshold (i .e . Ei = 10-50 ) can be chosen without any loss . Thus we shallnot report this value. In all the examples, rk denotes the residual computedrecursively by the algorithm.

Our numerical experiments seem to indicate that the global minimization isbetter than the step-by-step minimization . Thus, except in the example 1, weshall only give the results obtained by the global minimization .

Our heuristics for the jumps are sensitive to the choice of e and of 62 becausethe first value is related to the beginning of the jump and to its length and too

large a value can produce too long a jump . Moreover, when Qk+i~ < E, the

condition Irk+111 < E2 is tested and thus, too small a value of E2 could increasethe length of the jump up to the maximum allowed dimension .


195

The sensitivity of our heuristics is more important than in the algorithm pro-posed in [9] where the software CADNA [19, 20] insures that the tests for de-tecting numerical instabilities produce a quite stable program .EXAMPLE 5 .1 . The first example was proposed by Joubert [33] . He considered

the following matrix of dimension 4 that leads to a breakdown in the biconjugategradient method, for almost every starting vector x o

/ 1 -1 0

o \1

1 0

00 0 3 -10

0 1

3 /

For b = (0, 2, 2, 4), the system has the solution x = (1,1, 1, 1). If we takey = (1, 1, 1, 1), the Bi-CGSTAB breaks down at the iteration 2 . Our look-aheadversion, with r = 10-20 , jumps from no = 0 to n 1 = 2 due to a (l 1) = 0 and whenn3 = 4, we obtain, with the step-by-step minimization, IIr 3 II = 0.17 x 10

-13

and, with the global minimization, IIr 3 II = 0.33 x 10-14 . If we take y = r 0i theBi-CGSTAB, at the iteration 4, gives Ir4lI = 0.47 and the solution is obtainedonly at the iteration 7 with Ir7II = 0.13 x 10-15 . With e = 10 -6 , our programhas a jump from n1 = 1 to n2 = 3 due to Q21) < e and, for n3 = 4, we obtain,with the step-by-step minimization, IIr3II = 0.7 x 10 -14 and, with the globalminimization, IIr3II = 0.1 x 10 -14 .

EXAMPLE 5 .2 . Let us now consider the system (Gutknecht [29])

A=

the first iteration when n is even since (y, r0) _ (y, Aro) = 0 .For n = 400, x0 = 0, E = 10 --30 we have a jump of length 2 at the first iteration

to avoid the breakdown and we obtain n54 = 55 and IIr5411 = 0.66 x 10-11 whichcoincides with the actual residual (that is the residual computed from xk byrk = b - Axk) . When e = 10° 20 and E 2 = 10-14 , we have again the jump oflength 2 at the first iteration but we also have several jumps (from n34 = 35 ton35 = 37, from n35 = 37 to n36 = 40, from n36 = 40 to n37 = 42 and, finally, ajump from n37 = 42 to n38 = 48) and we obtain IIr38II = 0.42 x 10-14 and anactual residual of 0 .62 x 1014 .The behavior of the algorithm for n .= 400 and these two different values of

i s shown in Fig . 1 .

/2 1

\ /1\

/3\0 2 1 1 31 0 2 1 4

1

0

2 1 1 41

0 2/'\1/ \3/

For x0 = Q and y = (0,0.0, -1, 1, . . . , -(-1)r_1 , 0)T , a breakdown occurs at

196 C. BREZINSKI AND M. REDIVO-ZAGLIA

Fig . 1

nk

Figure 5.1 : Behavior of algorithm for system in Example 5.2, n = 400 andE= 10-20 , 10-30

EXAMPLE 5.3 . Let us now consider the following system (Brown [17])

When a = 0, a breakdown occurs every odd step of the Lanczos method . Ifwe take n = 400, a = 1 .1, x0 = 0 and y = r0, the Bi-CGSTAB converges tothe solution quite regularly and at the iteration 61, we have HHrsil1 = 0.84 x10-14 . With e = 10 -14 , our program detects some numerical instabilities in thecomputation and it jumps several time (the first one from n19 = 19 to n20 = 21) .When n26 = 40, 11r2611 = 0.37 x 10-13 which coincides with the actual residual .These results are presented in Fig . 2 .

a

1

\ /1\

/a+1\-1 a

1 1 a

1 a-1 a

-1 a 1 1 a-1 of \1/ \a-1J

Figure 5 .2 : Behavior of algorithm for system in Example 5.3, n = 400 and e = 10 -14

EXAMPLE 5.4 . Let us consider the n x n matrix


Fig . 2

197

Its determinant is equal to (1 - a,) (1 - a2 ) . . . (1 - a,_,) . Thus the matrix Ais singular if and only if ai = 1 for some i. With ai = 1 + i5, 6 = 10-2 , x 0 = 0,y = ro and n = 100, the Bi-CGSTAB does not converge and, at the iteration100, we get Ilriooll = 0 .4 x 10-2 . For E = 10-6 , our look-ahead program exhibitsseveral jumps (from n7 = 7 and with small values of Mk) and, when n46 = 100,we obtain IIr46IH = 0.6 x 10-8 which coincides with the actual residual. Theresults are shown in Fig . 3 .

1 1 1 . . . 1 1\a1 1 1 . . . 1 1

A= aI a2 1 . . . 1 1

~ al a2 a3 . . . a.-I 1

198

wO

OCO

OqO

Figure 5 .3: Behavior of algorithm for system in Example 5 .4, n = 100 and E = 10-6 .

6 Conclusions .

In this paper we

•

discussed a class of methods based on Lanczos polynomials. This class, calledBiCGM, contains the CGS and the Bi-CGSTAB as particular cases,

• gave a recursive algorithm for their implementation in the case where the auxil-iary polynomials Qk appearing in these methods satisfy a three-term recurrencerelationship,

• showed how to cure the breakdowns and near-breakdowns arising from the

polynomials Pk and P(Ii of the Lanczos method. The case of the Bi-CGSTABalgorithm was treated in details .A lot of work remains to be done on these methods . We have

• to investigate the theory of the methods of the BiCGM class and, in partic-ular, their convergence properties and the choice of the polynomials Qk. Morenumerical experiments with the methods of the BiCGM class proposed in section2 have to be conducted,

•

to study the heuristics for deciding when and how far to jump for avoidingnear-breakdowns. Our numerical results are quite sensitive to the choice of the


Fig . 3

LOOK-AHEAD IN BI-CGSTAB 199

various e appearing in the algor

s which leads to thin

at our strategyfor the jumps could be improved,

• to study, on a theoretical basis, the three m imization strategies we proposed .In particular, our line minimization assumes that the polynomials Qk are of thedegree nk and the formulae we gave are no longer valid for this minimization .However, as we saw above, it is quite easy to derive the recurrence relationshipsto be used when it is assumed that Qk has a degree qk < nk . But, whenthe difference nk - qk is large, these formulae need the computation of manymatrix-by-vector products or the use of AT and the method loses one of its

advantages . This is, in particular, the case when qk = k since, then, thedifference nk - k depends on the sum of the lengths of all the previous jumps,

• to study how to deal with the breakdowns and near-breakdowns arising fromthe polynomials Qk since our treatment is only able to cure those coming out

from the polynomials Pk and Pt'> . In particular, when sk = 0, our algorithmhas to be stopped since the systems giving the coefficients of the polynomialsEk are all singular .

These questions will be discussed in subsequent publications.

Acknowledgements .

We would like to thank Hassane Sadok for several interesting discussions . Thispaper was completed while the second author was an invited professor at theLaboratoire MASI of the Universite Pierre et Marie Curie in Paris . She wouldlike to thank Professor Rene Alt for his kind invitation and for providing her allthe working facilities. Finally, we also acknowledge the help of the referee whosecomments helped us to clarify some points and to improve the presentation ofthe paper .

REFERENCES

1. R. E. Bank and T. F. Chan, An analysis of the composite step biconjugate gradientmethod, Numer. Math., 66 (1993), pp . 295-319 .

2. R. E. Bank and T . F. Chan, A composite step bi-conjugate gradient algorithm forsolving nonsymmetric systems, Numerical Algor ., 7 (1994), pp . 1-16 .

3. B. Beckermann, The stable computation of formal orthogonal polyno

Is, Ann.Numer. Math., to appear .

4. C. Brezinski, Generalisations de la transformation de Shanks, de la table de Padeet de 1e-algorithme, Calcolo, 12 (1975), pp . 317-360 .

5. C. Brezinski, Pade-Type Approximation and General Orthogonal Polynomials,ISNM vol . 50, Birkhauser, Basel, 1980 .

6. C. Brezinski, CGM: a whole class of Lanczos-type solvers for linear systems, Publi-cation ANO-253, Universite des Sciences et Technologies de Lille, November 1991 .

7. C. Brezinski and A. C. Matos, Least-squares orthogonal polynomials, J. Comput .Appl. Math., 46 (1993), pp . 229-239 .

200


8 . C. Brezinski and M. Redivo-Zaglia, Hybrid procedures for solving linear systems,Numer. Math., 67 (1994), pp . 1-19 .

9. C. Brezinski and M. Redivo-Zaglia, Treatment of near-breakdown in the CGS al-gorithm, Numerical Algor ., 7 (1994) .

10. C. Brezinski, M . Redivo-Zaglia and H. Sadok, Avoiding breakdown and near-breakdown in Lanczos type algorithms, Numerical Algor., 1 (1991), pp. 207-221 .

11 . C. Brezinski, M . Redivo-Zaglia and H . Sadok, Addendum to "Avoiding breakdownand near-breakdown in Lanczos type algorithms", Numerical Algor ., 2 (1992), pp .133-136 .

12. C . Brezinski, M. Redivo-Zaglia and H. Sadok, A breakdown free Lanczos type al-gorithm for solving linear systems, Numer. Math., 63 (1992), pp . 29-38 .

13. C . Brezinski, M . Redivo-Zaglia and H. Sadok, Breakdowns in the implementationof the Lanczos method for solving linear systems, Comp. & Math . with Applics .,to appear .

14 . C. Brezinski and H . Sadok, Avoiding breakdown in the CGS algorithm, NumericalAlgor., 1 (1991), pp . 199-206 .

15 . C . Brezinski and H. Sadok, Some vector sequence transformations with applicationsto systems of equations, Numerical Algor ., 3 (1992), pp . 75-80 .

16. C . Brezinski and H . Sadok, Lanczos type methods for solving systems of linearequations, Appl. Numer. Math., 11 (1993), pp. 443-473 .

17. P. N. Brown, A theoretical comparison of the Arnoldi and GMRES algorithms,SIAM J. Sci . Stat . Comput., 12 (1991), pp. 58-78 .

18. T. F. Chan and T . Szeto, A composite step conjugate gradients squared algorithmfor solving nonsymmetric linear systems, Numerical Algor ., 7 (1994), pp. 17-32 .

19. J . M. Chesneaux, Stochastic arithmetic properties, in Computational and AppliedMathematics, C . Brezinski and U . Kulisch eds., North-Holland, Amsterdam, 1992,pp. 81-91 .

20. J. M. Chesneaux and J. Vignes, L'algorithme de Gauss en arithmetique stochas-tique, C.R. Acad . Sci . Paris, II, 316 (1993), pp . 171-176 .

21 . A. Draux, Polynomes Orthogonaux Formels. Applications, LNM 974, Springer Ver-lag, Berlin, 1983 .

22. H. C . Elman, Iterative Methods for Large, Sparse, Nonsymmetric Systems of LinearEquations, Ph.D. Thesis, Yale University, 1982 .

23. R. W. Freund, M. H. Gutknecht and N . M. Nachtigal, An implementation of thelook-ahead Lanczos algorithm for non-Hermitian matrices, SIAM J . Sci. Stat . Corn-put., 14 (1993), pp . 137-158 .

24. N. Gastinel, Analyse Numdrique Lineaire, Hermann, Paris, 1966.

25. B. Germain-Bonne, Estimation de la Limite de Suites et Formalisation de Procedesd'Accdldration de Convergence, These d'Etat, Universite de Lille I, 1978 .

26. G . H. Golub and D . P. O'Leary, Some history of the conjugate and Lanczos algo-rithms, SIAM Rev., 31 (1989), pp . 50-102 .

27. M. H. Gutknecht, The unsymmetric Lanczos algorithms and their relations to Padeapproximation, continued fractions, and the qd algorithm, in Proceedings of theCopper Mountain Conference on Iterative Methods, 1990 .


201

28. M. H. Gutknecht, A completed theory of the unsymmetric Lanczos process andrelated algorithms, part I, SIAM J. Matrix Anal. Appl., 13 (1992), pp . 594-639 .

29. M. H. Gutknecht, Variants of BICGSTAB for matrices with complex spectrum, IPSResearch Report 91-14, August 1991, Revised June 1992, to appear in SIAM J .Sci. Comput .

30. Cs . J. Hegedus, Generating conjugate directions for arbitrary matrices by matrixequations, I, II, Computers Math. Applic ., 21 (1991), pp . 71-85, 87-94 .

31. M. R. Hestenes and E . Stiefel, Methods of conjugate gradients for solving linearsystems, J. Res . Natl. Bur. Stand., 49 (1952), pp . 409-436 .

32. K. C . Jea and D. M. Young, On the simplification of generalized conjugate-gradientmethods for nonsymmetrizable linear systems, Linear Algebra Appl ., 52/53 (1983),pp. 399-417 .

33. W. Joubert, Generalized Conjugate Gradient and Lanczos Methods for the Solutionof Nonsymmetric Systems of Linear Equations, Ph.D. Thesis, University of Texasat Austin, Austin, 1990 .

34. M. Khelifi, Lanczos maximal algorithm for unsymmetric eigenvalue problems, Appl .Numer. Math ., 7 (1991), pp . 179-193 .

35. C. Lanczos, An iteration method for the solution of the eigenvalue problem of lineardifferential and integral operators, J. Res. Natl. Bur. Stand., 45 (1950), pp . 255-282 .

36. C. Lanczos, Solution of systems of linear equations by minimized iterations, J. Res .Natl. Bur. Stand., 49 (1952), pp . 33-53 .

37. W. R. Mann, Mean value methods in iteration, Proc. Am. Math . Soc., 4 (1953),pp. 506-510 .

38. B. N. Parlett, D.R. Taylor, Z .A. Liu, A look-ahead Lanczos algorithm for unsym-metric matrices, Math. Comput ., 44 (1985), pp. 105-124 .

39. M. Prevost, Stieltjes- and Geronimus-type polynomials, J. Comput. Appl. Math .,21 (1988), pp . 133-144 .

40. Y. Saad, Numerical Methods for Large Eigenvalue Problems, Manchester UniversityPress, Manchester, 1992 .

41. P. Sonneveld, CGS: a fast Lanczos-type solver for nonsymmetric linear systems,SIAM J. Sci . Stat. Comput ., 10 (1989), pp . 36-52 .

42. H. A. Van der Vorst, Bi-CGSTAB: a fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems, SIAM J. Sci . Stat. Comput .,13 (1992), pp . 631-644 .

43. D . M. Young and K . C. Jea, Generalized conjugate-gradient acceleration of non-symmetrizable iterative methods, Linear Algebra Appl., 34 (1980), pp . 159-194 .

Documents

Look-ahead in Bi-CGSTAB and other product methods for ...reichel/courses/intr.num.comp.2/spring14/lecture13/... · BIT 35 (1995), 169-201 . LOOK-AHEAD IN BI-CGSTAB AND OTHER PRODUCT