Algorithms for reproducing objects from their X-rays

COMPUTER CI:bLPHICS AND IMAC, E PROCESSING (1972) 1:(97-106)

Algorithms for Reproducing Objects From Their X-Rays *

N. T. GAARDE~ University of Hawaii

AND

G, T. HERMAN Depa'rtment of Computer Science

State University r York at Buffalo, Amherst, New York 14226

Communicated by A. ttosenfeld

Received July 30, 1971

A problem of considerable importance in biology is the reeonslxuetion of three-dimen- sional biological structures from electron micrographs. The problem is mathematically equivalent to that of reconstructing a three-dimensional object from X-rays taken at sev- eral angles. Solutions to these problems by Fourier methods have been proposed; unfor- tunately, the techniques are only applieahle to highly symmetxic objects; otherwise they require an unrealistic numher of electron micrographs. More recently, some Monte Carlo and algehraic reconstruction techniques have been proposed. They are applicable to asymmetric objects with a realistic number of eleclxon mierographs and less computer time; however, their perfbrmance with unknown objects is difficult to evaluate. In dais paper we discuss the lqrst steps towards linear-least-mean-squares techniques which can he used to develop rapid reconsta'uction algorithms that are also applicable to asymmet- ric objects wifla any number of electron micrographs. Such algorithms have the advan- tage ol" being the best of a class; in addition, their performance can be evaluated.

1. INTRODUCTION

The problem of reconstructing objects from daeir X-rays, or fi'om electron micrographs, can be reduced to the following mathematical problem. (A discussion of dais reduction process can be found, for example, in [8] and, in a briefer form, in [7].)

Let p be a function defined on pairs of integers (i,j), where 1 <~ i <~ n, i ~< j ~< n. Intuitively, we think of (i,j) as a grid point with x-coordinate i and y-coordinate j of a picture and p,z, the value of p at the point (i,j), as the grayness (darkness) of the picture at the point (i,j). We shall refer to p as a picture, and P~z as a picture element.

A project ion Pl~ is a partition of the points (i,]) into sets Rz~,~, Rlr,2 . . . . . a~

RI,,~.; (For 1 ~< l < l' -< = nk, R~r n Rl~.z,=~b, ~ Rk.z {(id)[l<-i<~n, 1<.

* An earlier version of this paper has been presented at the Conferencia International Sobre Sistemas, Redes y Computadoras, Mexico, 1971.

97

�9 Copyright 1972 by Academic Press, Ine.

98 G A A R D E R A N D H E R M A N

j ~< n}.) Intuitively, we think of a projection as a set of parallel lines drawn across file picture with those grid points which lie be tween two neighboring lines forming one of the Rz. The 1:l~,1, Re,2 . . . . . Ftk,.k will be referred to as the rays of Pk. Note that the mathematical formulation allows us to form projections and rays in ways essentially different fi'om the intuitive notion of parallel lines across the picture.

We assume that we have a fixed number, say m, of projections. For 1 ~ k m and 1 <~ I ~< n~, 4)k.t is the characteristic function of the l-th ray of the k-th projection, i.e.,

4)k'~(iJ)={~ i f ( i J ) otherwise, is in R,~.,,

Using this notation, and the definition of projections, we have that, for 1 ~< i ~< n, 1 ~<j ~< n, 1 < k ~< m,

@~.~(i,j) = 1, (i) z

and, for 1 ~< k < m, 1 ~< l, Io ~ nk, I # I0,

Notational comment: ~ stands to,. 2 and E stands for , Note that (2) l=1 i,J l=1 '=

is a consequence of (1). Now suppose that the only information we have available ahout a pic-

ture p is the sums of" gray levels in the various rays. That is, for i ~ k ~< m, l <~l<.n~,

= ( 3 )

and the value of rk,l is known to us for 1 ~< k ~< m, 1 < l ~< nk. I f ~ nir n 2 k

(which is usually the case in electronmieroscopy and X-ray photography), then Eqs. (3) do not uniquely determine p. The problem is to select from all the possible solutions which satisfy Eqs. (3) the one which is "most likely" to be the right one.

This last phrase indicates the most basic difficulty associated with the problem under consideration; it has no precise mathematical formulation. Clearly, what is required is a p which satisfies (3) and which optimizes some function f of the p~,j, but it is far from clear what that function f is.

In spite of this, there have been a number of attempts at a solution. A set of techniques, which have been around the longest, depend on obtaining from the ray-sums the values at selected points of the Fourier transform of the picture, inteqoolating to obtain a complete Fourier transform, and then estimate the picture by the use of an inverse Fourier transform. A compre- hensive survey of such techniques has been given in [5]. There are a number of objections to using these techniques to asymmetric ohjects (discussed in

REPRODUCING OBJECTS FROM THEIR X-RAYS 99

[7] and [8]). Some alternative Monte Carlo techniques have been proposed in [8]. These techniques aim at "smoothing" the original picture directly, either by averaging random solutions to (3), or by directly maximizing the function

- ~ . p , , j In p,.j. (4)

In [7] some algebraic reconstruction techniques are proposed which de- pend upon satisfying ray-sums one-by-one, without preserving the previous ones. The estimates for p provided by these techniques do not quite satisfy (3), but they are close, and in practice it was found that a good estimate can be obtained even from a small number of projections at a reasonable cost, These techniques have been used to reeonsmlct some ribosomes from electron micrographs in [1] and are further discussed in [6] and [10]. For completeness we mention [3], which deals with reconstruction when only two gray levels are allowed, and also [11-15].

In this paper we shall pave the way towards some techniques which have the following property: For any given set of projections, they have a large overhead }br an initial calculation, but once this is done, the reconstruction of any picture from the given set of projections is txivial, and costs very little. Since one can easily arrange that electron micrographs are taken at a predetermined set of angles, the overhead becomes insignificant in the long run.

2, MINIMIZING VARIANCE

In some sense, one may argue that the reconstruction which we want should be the "smoothest" possible of those satisfying (3). The reason for this is that we do not want our reconstruction to have features which are not a necessary consequence of the ray-sums. One way of trying to satisfy this criterion is the following:

"Find the p satisfying (3) such that

(p~,j - -ff)~ (5)

1 is mininmm." (ff is the mean of the p~.~. ff can be calculated as ~ 2 ~ r1~,z, for

l

all k. With experimental data the sum may depend on k, but in tlais paper we are assuming an ideal situation where each of the ray-sums are known exactly.)

Why is this a good way? One reason is the following: Projections more or less determine whether a given connected area should be dark or light. Suppose we divide the pictures up into connected areas X1, X2 . . . . . Xt. Suppose we know the mean gray level .~, in each of those areas. In order to get a smooth picture, we would want all points in these areas to have the same gray level. This is indeed what happens if we minimize (5), as can be seen below.

100 G A A R D E R A N D H E R M A N

t

t,J /=1 P$,jEXI

t

= 2 E 2 /=1 Pl,~ ~Xt

t = 2 ~ [(PI'J--~'I)sH-2(P'o--Xl)(:~Z--P) H- (:T:t -- #)'2].

/=1 Pio EXI

This is minimal if and is zero since ~ (Piz

only if p~,~ ----- fig, for Pl.J E Xg. (Note that the second term - ~l) = 0.) The somewhat quest ionable intuit ive argu-

Pt,j~ X ment above has been reinforced by experimental results in [7-10], where it was invariably found that visual smoothness of a picture satisfying con- straints of the type expressed in (3) is directly related to its variance. The value of (5) was smaller for the smoother pictures.

We now investigate how to obtain the required solution. First of all con- sider new variables

x~,j --- p~,j - ~, (6)

2 4)k,l(id)x~,j = Ck.z, (7) t,J

where ck,z = r~,t - P~,cg, where PJ~,z is the n u m b e r of points in/t1~,z. We want to find a solution to (7) which minimizes

J. (8) t,d

From a purely mathematical point of view, this problem has been solved. In fact it is a standard problem of minimizing a quadratic ~hnction subject to linear equality constraints. For a mathematical solution, see, for instance, [2, Section 2.5]. Unfortunately, the implementa t ion involves the inversion of a number os matTices and, unless some approximation techn ique is used, it is very costly. For example, in the fairly likely case of n -- 100, m = 10 a direct implementat ion would have a cost os several thousand dollars at present-day computer prices. It is therefore desirable to try to find another, cheaper technique.

Using Lagrange's method of unde t e rmined multipliers (see [4]), we define

F(x~z) = ~ xiz2 - 2 ~ ak,z [ i~ j ~bk~,(ij)x~,~- ck,z], (9)

where the ak.l are constants to be determined. If the x~.j satisfy (7) and mini- mize (8) then we get that

OF - - 2 2 r = O, (10)

Oxtd /ol

and so

xt,j = E as~,, qSk,t(id). (11) kd

REPRODUCING OBJECTS FROM THEIR X-RAYS i01

Substituting in (7)we get that, for 1 ~< ko ~< m, 1 ~< lo < nko,

2 r r o~,,.~ = Cko.,o. t,J 1r

(12)

(i3)

The value of the expression in the square parentheses is the number of points in which the/-th ray of the k-th projection and the 10-th ray of the k0-th projection intersect If k = ]%,

Pko.t~ if I = lo.

Let us use the notation

a(~,o,o~(k.,) = .~, 4'~,o.~o(id) 4,.~( id). (17) IO

From (14), (15), and (17) we get that, for i ~< ko ~< m and i ~< lo < nao,

7~,~,,,,,, ~,~0,~0 = C, ,o, ,o - ~, ~, a(ko.~o~(,,.,) ,~,~,~. (18) kr l

1[ O~k~176 : Pko,l'-"~o Ck~176 - -

Using (11) and (19) we get that

where

~ au~o.Zo)(1.z) o~,.z]. (19) k#ko l

x,,~ = ]2 r [c~,,, - m,,~], k,l ~)k,l

(20)

Intuitively, we can say that the required value of x~,j can be obtained by correcting the ray-sums ck.z by subtracting Yk,,, dividing the corrected ray- sums by the number of points in the ray in question, and then, ~br each ray containing the point (i,j), adding up the divided corrected ray-sums. The problem is that we do not yet have a method for working out the correcting term y~.z.

We shall now show that under certain assumptions the right result is ob- tained by taking yka = 0, for all k and l. These assumptions are such that it would be difficult to meet them in practice. In spite of this, our result will lead to an algorithm which can be uset:ul in practice. In particular, the as- sumptions will be valid if we had just a horizontal and a vertical projection.

Assumption 1. Any ray of any projection has exactly one element in com- mon with any ray of any other projection.

uk,, = E E "<~."(~o.'o)~ko.o. (2i) ko~k lo

102 G A A R D E R AND H E R M A N

Mathematical ly , this can he expressed as ibllows. For 1 < k ~ m, l ~ < k o ~ < m , k # k o , l~< l~<nt~and l<~lo<<-n~o ,

a~ko,Zo~(~,z~ = 1. (22)

Assumption 2. The n u m b e r of rays in each project ion and the n u m b e r of e lements in each ray are all equal to n, i.e., for 1 ~< k ~< n and 1 ~< l ~< nk,

n~ = P/c,z = n. (23)

From (21) and (22) it follows dmt

u,~., = y__, E ~,,0.,~ (9.4) k o ~ k lo

In otlaer words, Assumption 1 is sufllcient to insure that the correc t ing term is the same for all rays in any given projection, From now on let

YI~ = Ylr . . . . . YI~,,,~,. (9,5)

Assuming (23) and (25), we get another condi t ion on the y~r by suhst i tut ing (20) into (7)

qSk'z (i J)[lco~.lo _~:o,z. ( iJ)n (cl~~162 =ck'l"

From this we can argue as follows:

1 .r [(~,j ,b,~.z(id),b,,o,to(ij))(c,~~ 1 =c,~.t. But fi'om (17), (22), (23) and (2) we get that

n, if ko = k, and lo = l, ~,j 4)l~,z(i,j)dpl~..l,,(i,j) = O, i fko k, and lo r l,

1, if ko r k,

and so we obtain

(c~., - u~) + 1_ ~ , ~ (c~o,. - u~.) ~ c~,,. fl, ir162 lo

From (7), (1) and (6) it follows that

lo i , j l,J

Suhsti tut ing into (29) we get that

1 - u ~ + n ~ ( - n . u~)=0,

k o ~ k

and hence daat

Yl~ = 0. k

From (20) we obtain using (23) and (25) that

(26)

(27)

(28)

(29)

xi,~ = ~ , x~j = O. (30) hJ

(31)

(32)

REPRODUCING OBJECTS FROM THEIR X-BAYS 103

1

Using (1) and (82) we see that the second term is indeed 0 and so

1 x~,j = n ~ . epl . , ( i , j )c~, . (34)

k,l

This is the same as (20) with yk,z = 0. So there are situations in which the correcting tel-m yk,l in Eq. (20) is 0.

If we assume that this is so, we get from (6), (7) and (20) that

p~,j = p -t- S ~ H i ' J ) (rk,l -- Pk,t ' ~). (35) Pk.l

Using (1), this gives

( ~ d~k'l(tj) rl, Z)-- ( m - - 1 ) " ft. (36) P~'J ~--" Pl,',l

We must emphasize that the p described in (36) does not necessarily satisfy the constraints (3), but it does so under any set os assumptions which results in eliminating the effect the correcting terms yk,z in Eq. (20).

3, A FAST AND ROUGH ALGORITHM

We may consider the p described in Eq. (36) to be an approximation os the p which satisfies constraints (3) and minimizes (5). The advantage of doing this is that, quite clearly, there is a very fast algorithm to implement (36). The only problem is that the estimate obtained may be a very bad one.

In order to test this estimate, we carried out a number of experiments. Our object was a 49 • 49 digitized picture os a little girl, Judy, with 16 gray levels. Judy's picture has been used in previous publications [7, 8], and interested readers may have a look at it there. Also, the projections we took were the natural ones for object reconstruction; the rays were determined by parallel straight lines across the picture. This, of course, meant that the assumptions of fine last section were not satisfied.

We have found that the algorithm based on Eq. (36) is indeed very cheap to operate, An individual reconstruction of Judy had a cost in the order of 10-15 cents. However, the reconstructions were not as good quality as could be achieved by more expensive runs of any of the other methods. Also, the reconsta'uction, being a one-shot job, could not be improved by longer runs in the same way as it could be done by previous methods.

Another interesting point was that we found that 4 projections gave an optimal result. Using additional projections resulted in a reconstruction which was further away from the original. This is not altogether surprising, since the more projections are used, the more the assumptions of the last section are violated.

Experiments have been carried out when the 4 projections were taken at angles equally spaced in a range of ___15 ~ ___30 ~ and _-_60 ~ The reasons for

104 G A A R D E R A N D H E R M A N

studying small range of angles is connected with the nature of electron microscopes and is explained in [7]. The method descr ibed above was compared to the so-called direct additive method descr ibed in that paper. The table below shows the results of this comparison. The 8 in those tables is the Euclidean distance

[l~j~ 71,2

between the original picture /9 and reeonsh'ucted picture p'. All times are on the CDC 6400 and the direct additive method has been iterated five times.

Range a Time (in seconds)

Direct additive New method Direct additive New method

-+15 ~ 3.I 3.9 4.2 .7 ---30 ~ 2,7 3.4 4.9 .9 -60 ~ 2.4 3.0 6.0 1.1

It must be pointed out that by taking fewer iterations the time for the direct additive method can be decreased without the 8 exceeding the value of the new method. An improved version of the algorithm based on (36) incor- porating some of the ideas of [12, 147 15] is discussed in [9].

The main conclusion is that for the same quality of reproduction the method described in this section is cheaper to implement than the direct additive method which has until now been the cheapest picture reconstruction technique. However, the reconstructed picture shows only the more obvious features of the original and cannot be improved by additional projections or longer computer runs.

4. MORE ACCURATE ITERATIVE ALGORITHMS

Equation (34) gives a solution to the picture reconstruction problem in a special case. In this solution the value xi,j is obtained as a linear combination of the ray-sums ck,z. However, the particular linear combination is not the one we want in genera], as it has been demonsb'ated in the last section. An alternative approach is to look for coefficients a~;fi (i ~< i,j ~ n, 1 ~< k ~ m, 1 ~< I ~< nl~) such that if we define

x u = ~ i'J' (37) Oglr Ic,l k,l

where cl, t is defined by (7), then the average value of

.~ (xi,j- x~,j)", (38)

taken over all pictures will be minimal. Note that if the a~;,~z are fixed, then since cjr is a fnnetion of the x~,j, (38) is a fimetion of the x u.

REPRODUCING OBJECTS FROM THEIR X-RAYS 105

We generalize the ahove notions in the following way. We require an algorithm, so that, given

(I1) A number n; (I2) An n 2 • n" symmebie non-negative definite matTix C~. (the estimated

covarianee mahix of x); (I3) A set of m projections, with nk rays in the k-th projection, on the set

{(i,j) ] 1 ~< i ~ < n, 1 ~<j ~< n},

the algorithm wilt give a set of numbers ak.~i'~ (1 ~ k ~< m,1 ~< 1 ~< n~r such that if 2~,j is defined by (37) with c~.~ defined by (7), then

E[I~ - xl"] (39)

is smaller than it would be with any other w~lues for the ai;!z. (For definition o t : E [ I ~ - x["], see [16]. Intuitively, E is the average error in the reconsh'uc- tion, dependen t on the assumed covarianee.)

If we can find such an atdz, then, on average, the procedure of using Eq. (37) to obtain a picture from its projections will be the best. It is a fortzmate and interesting faet, although not exactly trivial to prove, that the ~.~ ob- tained in this way will also satisf) Eq. (7), i.e., the ray-sums will be the cor- rect ones. Also, if C~ is the identity, the Yq.j will also minimize (8). Thus we have a solution to the problem of Section 9..

Clearly, Eq. (37) is about as eheap and easy to implement as the algorithm in the last section, provided that we know the a)!z. However, finding al,..liJ is a joh which needs to be done only once for a particular ray-pattern. Since it is easy to keep the ray-pattern fixed for eleetromnieroseoping or X-raying of various objects, finding the ~){l is really a one-time overhead.

We have deve loped an iterative method for finding ak.!l (or a close ap- proximation) which is guaranteed to converge. It requires storage propor- tional to n*, but the rate of eonvergence appears to be good. Since we are just in the process of implement ing this technique, we shall delay details until a later publication where we can report on experimental results.

ACKNOWLEDGMENTS

This research has been partially supported by NSF grants GJ 596 and GJ 998. Most of the programming has been done by Dr. R. Gordon and B. Weiss and J. Rowe. We are also grateful to the above mentioned individuals, as well as to A. Walker and S. Rowland for helpful discussions.

REFERENCES

]. R. BENDER, S. H. BELLMAN AND Ft. GOBDON, ART and the ribosome: A preliminary report on the three-dimensional structure of individual ribosomes determined by an algebraic reconstruction teehnlque, J. Them'. Biol., 9.9, 1970, 483-487.

2. J. C. G. BOOT, Quadratic Programming, Nm• Amsterdam, 1964. 3, S, K. CHANG, AND G. L, SHELTON, Two algorithms for multiple view 1)inary pattern re-

construction, IEEE Trans. on Systems, Man and Cybernetics, 1, 1971, 90-94, 4. R. COm~AN'r, Differential and Integral Calculus, Vol. 2, Interseience, New York, 1936. 5. R. A. CROWTHI';R, D, J, DEROSIER AND A. KLUG, The reconstruction of a three-dirnensional

structure from projections and its applieation to electron microscopy, Proc. Roy, Soc. London A 317, 1970, 319-340.

106 GAARDER AND HERMAN

6. G. FI:~IEDER AND C. T. HERMAN, Resolution in reconstructing objects from electron micro- graphs, J. Them'. Biol. 33, 1971, 189-211.

7. R. GORDON, R. BENDER AND C. T. HERMAN, Algehraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography, ]. Theor. Biol. 29, 1970, 471-481.

8. R. GORDON AND G. T. HERMAN, Reconstruction Of pictures fl'om their projections, Comm. ACM. 14, 1971, 759-768.

9. O. T. HERMAN, Two direct methods [br reconstructing pict~res fimn their projections: A cmnparative study, Comp, Graphics Ira. Pvoc., 1.

I0. G. T. HERMAN AND S. ROWLAND, Resolution in ART: An experimental investigation of the resolving power of an algebraic picture reconstructima technique.]. Theor. Biol. 33, 1971, 213-223.

11. G. N. I~AMACHANDRAN, Reconstmctlon of suhstanee from shadow, I. Mathematical theory with applicatima to three-dimensional radiography and electron microscopy, Proc. Indian Acad, Sci. i971,

12. C. N. RAMACHANDRAN ANI) A. V. LAKSHMINAI~AYANAN, Three-dimensional reconstruction from radiographs and electron micrographs, II. Application of conwfiutions instead of Fourier transforms, Proc. Nat. Acad. Sci. U. S. A., 68, 1971, 2236-2240.

13. G, N. ]~.AMACHANDRAN AND A. V. LAKSHMINARAYANAN, Three-dimensional reconstruction from radiographs and electron microgmphs, III. Description mad applicatima of the con- volution method, Indian J, Pure Appl. Phys., Raman Memorial Issue, to appear.

t4. B. K. VA1NSHTEIN, Finding the structure of objects from projections, Sou. Phys. - Crystal- logr., 15, 1971, 781-787 (translated from Kristallografiya, 15, 1970, 894-902).

15. B. K. VAINSHTEIN, The synthesis of projecting functions, Dokl. Akad. Nauk SSSR 196, 1971, 1072-1075.

16. A. M. YAGLOM, An Introduction to the Theory of Stationary Random Functions, Englewood Cliffs, N. J., Prentice-Hall, ]962.