An anisotropic nonlinear diffusion approach to image interpolation

S.Morigi‡, F.Sgallari‡‡Dept. Mathematics, University of Bologna, Italymorigi,sgallari @dm.unibo.it

C. Zannoni\\CINECA, Casalecchio di Reno, Bologna, Italy

c.zannoni@cineca.itA. Cappello†

†D.E.I.S, University of Bologna, Italyacappello@deis.unibo.it

Keywords: Image interpolation, PDE, Image Processing.

Abstract

In medical imaging, a three dimensional (3D) object geom-etry together with its density features must often be recon-structed from serial cross sections. Usually, these cross sec-tions are not evenly spaced and are affected by noise in-trinsically linked to the acquisition method. In this paperwe present a new approach to the image interpolation prob-lem based on anisotropic nonlinear diffusion. The methodfilters the anisotropic noisy data and processes them to geta 3D isotropic data reconstruction of geometry and densityfeatures. Computational results with syntetic and computedtomographic (CT) real images are presented and discussed,focusing on X-Ray examinations of long bones.

1 Introduction

Image interpolation is an important technique in the field ofthe 3D reconstruction from cross sectional images. Medicaltechniques such as CT, magnetic resonance imaging (MRI)or direct sectioning, for example, generate a series of cross-sectional images from which 3D object geometry and densityfeatures have to be extracted and reconstructed. Usually, thecross sections are not evenly spaced in order to capture themost significant parts of the object, the number of scans islimited to reduce the radiation dose delivered to the patient,moreover, these images are often affected by the noise intrin-sically linked to the acquisition method. For improving vi-sualization and 3D-feature enhancement and reconstruction,the obtained anisotropic acquired data set needs to be filteredand interpolated to a isotropic image data set.

The aim of this work is to present a new approach to theimage interpolation problem based on anisotropic nonlineardiffusion. The proposed nonlinear Partial Differential Equa-tion (PDE) models, representing a multiscale analysis, fil-ter and interpolate the sequence of slices preserving and en-

hancing structures and converting the acquired data set intodata of uniform discretization. PDE models and methodshave recently emerged in image processing as an alterna-tive to traditional transform and statistically based methods[1, 5, 21, 22]. The application of traditional pre-processingalgorithms (moving average, median and gaussian filtering)reduces the noise superimposed to the image but does notpreserve a good definition of the details. The PDEs aredesigned to possess certain desirable geometrical propertiesand allow a more systematic approach for restoring imageswith sharp edges and segmentation. The preliminary resultswe have obtained investigating the use of a PDE approachto the image interpolation and reconstruction problem lookpromising even if some more work still needs to be done toget a suitable PDE model for the problem.

The paper is organized as follows. Section 2 introducessome basic notations together with a problem formulation.Section 3 presents classical interpolation techniques, appliedeither on slices either on objects. A brief introduction on theapplication of PDE models to image processing is presentedin Section 4, while Section 5 provides a description of thePDE methods proposed for the interpolation problem. Sec-tion 6 presents some experimental results of using the pro-posed methods to process syntetic and CT cross sectionalimages.

2 Notation and problem statement

The input image sequence we need to elaborate can be con-sidered both as avolume (or 3D image) and as a single twodimensional (2D) image evolving in time (image sequence).

A volume can be represented by a real functionu0(x), u0 :Ω → IR, whereΩ ⊂ IR3 represents a spatial domain. A 2Dimage sequence can be modeled by a real functionu0(x, θ),u0 : Ω × [0, T ] → IR, whereΩ ⊂ IR2 represents a spatialdomain, and[0, T ] the time interval.

A slice is equivalently represented byu0(x1, x2, x3), withconstantx3 in the 3D image notation, and byu0(x1, x2, θ)for a constantθ value, in the 2D image sequence.

The input data set we wish to interpolate consists of a se-quence ofn slices. Interpolation transforms a volume (or 2Dimage sequence)u0, into another volume (or 2D image se-quence)u, composed bym > n slices. After interpolation,in fact,m−n new slices will appear in between the adjacentinput ones.

3 Related work

Traditional interpolation techniques can be divided into twogroups: scene-based and object-based. Scene-based meth-ods determine pixel values for interpolated cross-sections di-rectly from the density values of given slices. In object-basedmethods, some object information extracted from the slices,such as shape, are used in guiding the interpolation process.We should note that none of the methods belonging to thesetwo classes deals with the denoisying problem.

The most commonly used scene-based methods are thestraightforward nearest-neighbor and linear interpolation[14, 6]. Higher order techniques such as spline-based im-age interpolation have been used to fill the interslice spaces[12], obtaining slightly superior results. However, if thecross-section morphology of individual structures changes oris displaced significantly from slice to slice, image intensi-ties from one tissue type may be interpolated with those ofanother, producing artifacts in the images and in the result-ing 3D reconstruction (see Figure 2 in Section 6). Hence,image interpolation methods of this class often do not pro-vide an appropriate correspondence between structures of thesame tissue type in adjoining slices, especially near the ob-ject boundaries. This problem is accentuated as the distancebetween adjacent slices increases. Summarizing, this classpresents three major disadvantages: low frequency informa-tion is introduced, artifacts appear, structures and anatomy ofthe object tend to be lost. Thus, when objects are extractedand displayed from the interpolated image, these objects of-ten exhibit a staircase and generally unsatisfactory appare-ance.

Object-based interpolation methods offer an improvementover scene-based interpolation [3, 13, 15, 7, 8]. Raya andUdupa in [18] have introduced the shape-based interpolationtechnique that has become the most relevant of the object-based interpolation class. These methods apply the distancetransform to the grey-level images, then interpolate the dis-tance maps between the slices and finally convert back intogrey-level images to get the interpolated slices. The imagespace is interpolated by incorporating the correspondencebetween objects on adjacent slices that express changes inthe shape and position. Recently, shape-based interpolationmethods have been improved using distance transform andmorphing [4]. Methods belonging to the object-based in-terpolation class preserve the anatomy and structures of theobjects and solve the artifact problems. However, a funda-mental problem with each of these techniques is that eachcomputation is very time consuming. Moreover, they don’tuse the original gray-scale information and have difficulties

with images containing three-like structures (see Figure 3 inSection 6) .

Our methods can be considered an example of object-based method that works directly on the density values of theobjects. The method will be applied to a sequence of CT im-ages of a femur bone. These tomographic slices present thefollowing peculiarities: adjacent slices differ only for slightchanges due to translations, rotations and scaling. This al-lows us to search for correspondence between points onlyover a small neighbor of the reference point. Moreover, re-gions in consecutive slices have quite similar shape and area,so big variations can be only due to errors in the choice ofthe scanning sequence.

4 PDE for image processing

The main reason for using nonlinear diffusion filtering in im-age processing is provided by the very good results knownin the literature in many fields of application. Unlike lin-ear diffusion filtering (which is equivalent to convolving theoriginal image with a Gaussian function) edges remain welllocalized and can even be enhanced. For mathematical fun-damentals and an extensive literature on this class of filterswe refer the reader to [21, 23].

PDEs turn to be the appropriate mathematical tool thatleads to well-defined and powerful algorithms for prepro-cessing of raw data. In particular, PDEs encode adaptive be-haviour in a purely data-driven way that is flexible enough tocope with the rich image structure commonly found in med-ical images.

Nonlinear PDE-based image processing has been intro-duced to the field of computer vision by Perona and Malik[17] and after them applied to the field of medical imaging(see [22, 19] and the references therein) and studied in theo-retical and computational details (see [2, 1, 5]).

The image multiscale analysis, as it has been introducedin [1], associates withu0 a family u(t, x, θ) of smoothed -simplified images (or a family of smoothed sequences) de-pending on an abstract parametert, thescale. As it has beenproved in [1], if such family fulfills basic assumptions - pyra-midal structure, regularity and local comparison principle -then, it can be represented as a solution of a second orderparabolic PDE with the initial condition given by the origi-nal noisy imagesu0(x, θ).

In this paper we propose to process a single image (2D/3D)using the model due to Alvarez et al. in [2] represented bythe nonlinear PDE:

∂u∂t

= g(|∇Gσ ∗ u|)|∇u|∇.(∇u|∇u|

), (1)

while the model we consider for 2D image sequences (rep-resenting the 2D slicesθi, i = 1, . . . , m of the volume), hasbeen introduced in [19], and is given by

∂u∂t

= clt(u)∇ · (g(|∇Gσ ∗ u|)∇u). (2)

These equations are accompanied with zero Neumannboundary conditions on the spatial boundary and suitableboundary conditions in time.

The parameters in (1-2) fulfill the following assumptions

• g is a continuous function,g(0) = 1 and0 < g(s) → 0 for s →∞

• Gσ ∈ C∞(IRN ) is a smoothing kernel,N = 2, 3,∫

IRN Gσ(x)dx = 1, Gσ(x) → δx for σ → 0, δx - theDirac measure at pointx.

Common choices for the two previous parameters are e.g.

g(s) =1

1 + K ∗ s2 (3)

with some constantK, andN -dimensional Gauss function

Gσ(x) =1

(2√

πσ)N e−|x|24σ . (4)

The first equation (1) is related to the so calledmorpho-logical multiscale analysismodels and it has the peculiarityof preserving image geometrical features like silhouettes inthe smoothing process given by diffusion [2].

Notice that the caseg(s) ≡ 1 reduces equation (1) to thewell-known level set equationproposed by Osher & Sethian[16, 20]. Thelevel set equationmoves each level set ofu(namely, level line in 2D and level surface in 3D) in the nor-mal direction with velocity proportional to its mean curvaturefield. Moreover, it yields the so calledmorphological princi-ple: if u is a solution then, for any nondecreasing functionϕ,ϕ(u) is a solution as well.

The term|∇u|∇.( ∇u|∇u| ) in (1) represents a degenerate dif-

fusion term, which diffusesu in the direction orthogonal toits gradient∇u, and does not diffuse at all in the direction of∇u. The functiong(s) depending on|∇Gσ ∗ u| is an edgeindicator and is used for the ”enhancement” of the edges.

The second equation (2) was proposed in [19] and ap-plied to 3D echocardiographic image sequences in orderto consider a time coherence of successive frames. Theproposed model combines ideas of the regularized Perona-Malik anisotropic diffusion model [17] and Galilean invari-ant movie multiscale analysis equation of Alvarez, Guichard,Lions and Morel [1]. It filters the space-time image sequencewhile respecting the spatial as well as temporal coherentstructures. The assumptions considered in [19] are that thestructures in the image are formed by points that preservetheir intensity along the motion trajectory. Such objects arecalled Lambertian structures. Moreover it is assumed thatmotion is Galilean locally in time, i.e. the trajectories aresmooth.

The termclt(u) has been presented by Guichard in orderto respect the Lambertian structures; the nameclt, in fact, in-dicates the relation to the curvature of Lambertian trajectory.Guichard in [1, 10] proposed the following quantity:

clt(u) = minw1,w2

1∆θ2 (| < ∇u, w1 − w2 > |+ (5)

|u(x− w1, θ −∆θ)− u(x, θ)|+|u(x + w2, θ + ∆θ)− u(x, θ)|)

wherew1, w2 are arbitrary vectors inIRN , ∇u denotes par-tial derivatives with respect to all space variables and∆θ isthe time increment.

The scalar functionclt(u) introduces a measure of coher-ence in time for the moving structures, in such a way thatlow values ofclt(u) correspond high coherence in time. Thevalue of clt(u) is small for the so-called Galilean trajecto-ries, i.e. for Lambertian points moving by a constant velocityfield. On the other hand, for noisy points, there is no motioncoherence and thus the value ofclt(u) will be large there. Itconsists of the sum of three positive parts and we want to findthe minimum in all possible directionsw1, w2. The last twoterms in the sum on the right hand side of (5) are related tothe differences in the intensities of end-points of candidateLambertian velocity vectorsw1, w2. To find the directionsof such vectors we look at the points which have the closestintensity value to the intensityu(x, θ) in the previous frame(term |u(x − w1, θ −∆θ) − u(x, θ)|) and in the next frame(term |u(x + w2, θ + ∆θ) − u(x, θ)|). Note that, if we findcorresponding Lambertian points both terms vanish. The firstterm in the sum, namely| < ∇u,w1−w2 > |/(∆θ)2, corre-sponds to the so calledapparent acceleration, i.e. to the dif-ference between candidate Lambertian velocity vectorsw1

andw2 in the direction of∇u. For details we refer to [1].The spatial smoothing in equation (2) is obtained by a

slight modification of the well-known Perona–Malik equa-tion called alsoanisotropic diffusionin computer vision com-munity. It selectively diffuses an image in the regions wherethe signal is of constant mean in contrast with those regionswhere the signal changes its tendency. Such diffusion pro-cess is governed by the shape of functiong and by its depen-dence on∇u which is in a sense an edge indicator [17].

Catte et al. introduced in [5] the convolution with Gaus-sian kernelGσ and they have established the well-posednessof the mathematical formulation.

5 Outline of the proposed method

Applying PDE models to solve the image interpolation prob-lem, we require to preserve points moving by a constantvelocity field through the slices and keeping their intensityalong the motion trajectory. In the spatial variables, that isin each slice, the smoothing process is desirable to conserveedges and filter out the noise and the artifacts which appearin the initial reconstructed images due to the linear interpo-lation process.

To respect these requirements we have chosen the non-linear degenerate diffusion equation of mean curvature flowtype (1) for the processing of the entire volume, and PDEequation model (2) for the processing of the 2D image se-quence.

In input our methods consider a sequence of imagesu0(·, θ), θ = 1, . . . ,m, with m the total number of slices;

Figure 1: The original slices (1-9) and the interpolated slices(1-9) between the slices (1,9) of the truncated cone volume,using PDE model (2).

this sequence is considered as a volume for model (1) andas a sequence of 2D images for model (2). Some of theseslices are the input data set, and we will refer to these as ref-erence slices, while in between we consider the slices com-ing from a trivial interpolation method, such as, for example,nearest-neighbor or linear interpolation. We suppose that,under a certain resolution, the shape of objects should changesmoothly. The proposed algorithm processes the 2D imagesequence (or volume) and is able to distinguish the noisefrom the contours of the different structures by using in a dis-tinct way spatial and sequence coherence. In this approach,correspondence is found between points in consecutive refer-ence slices, and then this correspondence is used to estimatedata between the slices. The method takes advantage of thefact that consecutive slices have small geometric differencesand carries out the search in predicted small neighborhoods.

We consider the model equation (2) where theclt(u) hasbeen modified as follows:

clt(u) = minw1,w2

1(α∆θ)2

(| < ∇u,w1 − w2 > |+ (6)

|u(x− w1, θ − α∆θ)− u(x, θ)|+|u(x + w2, θ + α∆θ)− u(x, θ)|)

whereα is used to choose as previous and next slices onlythe reference slices. Moreover, we added to the right-handside of equation (2) the term

f(u− ur)

whereur represents the nearest reference slice tou andfis a Lipschitz continuous, non-decreasing function, such thatf(0)=0. By means off on the right-hand side of (2), thesolutionu is forced to be close tour.

The discretization of the presented methods is based onsemi-implicit approximation in scale and finite volume inspace [19, 11]. Such discretization leads to the solution oflinear systems of equations for each slice of the sequence ateach discrete scale step. This can be done efficiently by usingpreconditioned iterative methods [19].

6 Examples

Experiments were performed on synthetic and real sequencesof images. The synthetic sequence was selected to presentthe features of the algorithm. First the PDE model (2) wasapplied on slices of size64× 64 that contain circular cuts ofan object that can be described as a truncated cone (see Fig-ure 1). We used a scale step of size0.01, and we looked ata 5 × 5 neighborhood to computeclt(u). There are 7 inter-polated slices generated. The 9 first images starting from thetop left corner are the original noisy images, while the other9 images are the result after 6 scale steps of the proposedPDE method (2), with initial conditions given by the resultof a simple linear interpolation between slices 1 and 9.

(a)

(b)

Figure 2: 3D display of interpolated slices: (a) original dataset (b) grey-level linear interpolated data set.

In the following experiments we have applied the PDE ap-proach to anin vitro acquired CT sequence of a human femur

selecting a subset ofn scan locations upon the set ofm avail-able slices. This would simulate thein vivo investigationswhere the number of CT scans has to be limited because itdetermines the quantity of radiation absorbed by the patient[24]. The 3D data set was obtained by scanning the cadaverfemur in vitro. The femur was positioned with the longi-tudinal axis parallel to the CT scanning direction and wasscanned with a 1mm step on a GE Sytec 3000 scanner. Thus,the 3D data set consists of 420 CT images corresponding tothe femur longitudinal dimension of 420mm. Each slice is a512× 512 gray level image.

The first experiment on the CT images shows isosurfacevisualization of the femur data set. The results displayed inFigure 2, show the original data set of the femur (Figure 2(a))and the reconstructed data set using linear interpolation usingstep 10 (see Figure 2(b)). Both data sets consist of 200 slicesof size256×256 with pixel grey levels in the range[0, 4095].

The second CT sequence of slices (1-4) is presented in Fig-ure 3. We show in Figure 4 the result of applying shape-basedinterpolation algorithm using slices (1,4) as reference slices.Notice the lost of gray-level details in the reconstructed slicesin between. The result of processing PDE model (1) on thesame data set is shown in Figure 5 after 4 scale steps.

The third experiment on real CT sequences is illustratedin Figure 6 and Figure 7. Reconstructed slices using PDEmodel (2) are illustrated in Figure 6, while the reconstructionobtained by shape-based interpolation is shown in Figure 7.

In Figure 8 results are shown from the reconstruction of apart of the femur bone using 8 reference slices equally spacedfor reconstructing a sequence of 40 slices. The 3D display ofisosurfaces has been obtained by shape-based interpolation(Figure 8(a)), by linear interpolation (Figure 8(b)), and bythe PDE model (2) (Figure 8(c)). Significantly smoother andless ”staircasing” representations of the femur bone are pro-duced using PDE models than those created using linear in-terpolation. Finally we should notice that these PDE methodsturn to be more efficient than the computationally expensiveshape-based interpolation techniques. The efficiency of thelatter technique is function of the number of different graylevels in each slice, while the PDE approach is influencedonly by the dimensions of the slice. In these preliminary ex-perimentations we have noticed that PDE methods performbetter 20 times than the shape-based ones.

7 Conclusion

In this paper, we have dealt with the important problem ofimage sequence interpolation by applying the PDE basedmethodology, which has been already proved to be very suc-cessful in anisotropically restoring images and segmentation.Our approach allows us to deal simultaneously with the prob-lem of denoising and interpolation. Results demonstrate thatthe algorithm provides good estimation of interpolated datafrom a visual qualitative point of view. Future work will

1 4

2 3

Figure 3: Original data set.

2 3

Figure 4: Interpolated slices: shape-based interpolation.

2 3

Figure 5: Interpolated slices: PDE model (1) interpolation.

Figure 6: Interpolated slices: PDE model (2) interpolation. Figure 7: Interpolated slices: shape-based interpolation.

(a)

(b)

(c)

Figure 8: 3D display of interpolated slices: (a) shapebased interpolation (b) grey-level linear interpolation (c)PDE model (2) interpolation.

investigate a serious comparison of the performance of themethods on the basis of quantitative analysis, according tothe common figures of performance merit [9].

References

[1] L. Alvarez, F. Guichard, P.L. Lions, J.M. Morel,Ax-ioms and Fundamental Equations of Image Processing,Arch. Rat. Mech. Anal. 123, pp. 200–257, 1993.

[2] L. Alvarez, P.L. Lions, J.M. Morel,Image selectivesmoothing and edge detection by nonlinear diffusion II,SIAM J. Numer. Anal., 29, pp. 845–866, 1992.

[3] W.A.Barret, R.R. Stringham,Shape-based interpola-tion of gray-scale serial slice images, SPIE Proceed-ings, 1898, pp. 105-115, 1993.

[4] L. Bin, E.R.Hancok,Slice interpolation using the dis-tance transform and morphing, in Proc. 13th Intern.Conf. on Digital signal processing, 2, pp.1083–1086,1997.

[5] F. Catte, P.L. Lions, J.M. Morel, T. Coll,Image selec-tive smoothing and edge detection by nonlinear diffu-sion, SIAM J. Numer. Anal., 29, pp. 182–193, 1992.

[6] A.Goshtasby,D.A.Turner, L.V. Ackerman,Matching ofTomographic slices for interpolation, IEEE Trans. OnMed. Imaging, 11, pp. 507-516, 1992.

[7] G.J.Grevera, J.K.Udupa,Shape-based interpolation ofmultidimensional grey-level images, IEEE Trans. OnMed. Imaging 15(6), pp. 881-892, 1996.

[8] G.J. Grevera, J.K. Udupa,Shape-based interpolation ofnD gray scenes, SPIE Proc., 2707 Medical Imaging ,pp. 106–116, 1996.

[9] G.J. Grevera, J.K. Udupa,An objective comparison ofinterpolation methods, IEEE Trans. Medical Imaging,17 , pp. 642–652, 1998.

[10] F. Guichard, Axiomatisation des analyses multi-echelles dimages et de films, PhD Thesis UniversityParis IX Dauphine, 1994.

[11] A. Handlovicova, K. Mikula, F. Sgallari,VariationalNumerical Methods for solving nonlinear diffusionequations arising in image processing, submitted,2000.

[12] G.T. Herman, S.W.Rowland, M.M.Yau,A comparativestudy of the use of linear and modified cubic spline in-terpolation for image reconstruction, IEEE Trans. OnNuclear Science, NS-26(2),pp. 2879-2894, 1979.

[13] G.T.Herman,J.Zheng, and C.A.Bucholtz,Shape basedinterpolation, IEEE Computer Graphics and Applica-tions, 12(3), pp. 69-79, 1992.

[14] W.E.Higgins,B.E.Ledell,A non linear filtering ap-proach to the gray-scale interpolation of 3D medicalimages, SPIE Proc. Image Processing 2167, pp. 284-295, 1994.

[15] R.A.Lotufo, G.T.Herman, J.K.Udupa,Combiningshape-based and gray-level interpolations, Visualiza-tion in biomedical computing, SPIE Proc. Visualizationin Biomedical Computing 1808, pp. 289-298, 1992.

[16] S. Osher, J. Sethian,Front propagating with curvaturedependent speed: algorithms based on the Hamilton-Jacobi formulation, J. Comput Phys., 79, pp. 12–49,1988.

[17] P. Perona, J. Malik,Scale space and edge detec-tion using anisotropic diffusion, IEEE Trans. PatternAnal.Mach. Intell., 12, pp. 629–639, 1990.

[18] S.P.Raya, J.K.Udupa,Shape-Based interpolation ofMultidimensional Object, IEEE Trans. On Med. Imag-ing, 9 (1), pp. 32-42, 1990.

[19] A. Sarti, K. Mikula, F. Sgallari,Nonlinear multiscaleanalysis of 3D echocardiographic sequences, IEEETrans. Medical Imaging, 18, pp. 453–466, 1999.

[20] J.A. Sethian,Numerical algorithm for propagating in-terface: Hamilton-Jacobi equations and conservationlaws, J. Diff. Geom., 31, pp. 131–161, 1990.

[21] J.A. Sethian,Level Set Methods and Fast MarchingMethods. Evolving Interfaces in Geometry, Fluid Me-chanics, Computer Vision, and Material Science, Cam-bridge Monographs on Appl. and Comp. Mathematics,Cambridge University Press, 1999.

[22] J. Weickert, A review of nonlinear diffusion fil-tering, in Scale-Space Theory in Computer Vision(B.t.H.Romeny, L.Florack, J.Koenderink, M.Viergevereds.), Lecture Notes in Computer Science 1252,Springer, pp. 3–28, 1997.

[23] J. Weickert,Anisotropic diffusion in computer vision,Teubner-Stuttgart, 1998.

[24] C.Zannoni, A.Cappello,M.Viceconti,Optimal CT scan-ning for long bone 3D reconstruction, IEEE Transac-tions on Medical Imaging, 17 , pp. 663–666, 1998.