Mathematics and Visualization

Series EditorsGerald FarinHans-Christian HegeDavid HoffmanChristopher R. JohnsonKonrad PolthierMartin Rumpf

Editors

ABC

Lars LinsenHans HagenBernd Hamann

Visualization in Medicineand Life Sciences

7With 76 Figures, 4 in Color

ISBN-13

This work is subject to copyright. All rights are reserved, whether the whole or part of the material isconcerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publicationor parts thereof is permitted only under the provisions of the German Copyright Law of September 9,1965, in its current version, and permission for use must always be obtained from Springer. Violations areliable for prosecution under the German Copyright Law.

Springer is a part of Springer Science+Business Media

c©

The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply,even in the absence of a specific statement, that such names are exempt from the relevant protective lawsand regulations and therefore free for general use.

A E

Cover design: design & production GmbH, Heidelberg

Printed on acid-free paper 5 4 3 2 1 0

springer.com

Typesetting by the authors and SPi using a Springer LT X macro package

46/SPi/3100

Jacobs University BremenTechnische Universität Kaiserslautern

P.O. Box 750561

School of Engineering and Science

Department of Computer Science

E-mail: hagen@informatik.uni-kl.de

One Shields AvenueUniversity of California

E-mail: hamann@cs.ucdavis.edu

Library of Congress Control Number: 2007935102

978-3-540-72629-6 Springer Berlin Heidelberg New York

Springer-Verlag Berlin Heidelberg 2008

SPIN: 12066520

Lars Linsen Hans Hagen

Bernd Hamann

Mathematics Subject Classification: 68-06, 68U05

67653 Kaiserslautern, Germany

28725 Bremen, Germany

Davis, CA 95616-8562, U.S.A

E-mail: l.linsen@jacobs-university.de

Preface

Visualization technology has become a crucial component of medical and bio-medical data processing and analysis. This technology complements tradi-tional image processing methods as it allows scientists and practicing medicaldoctors to visually interact with large, high-resolution three-dimensional im-age data. Further, an ever increasing number of new data acquisition meth-ods is being used in medicine and the life sciences, in particular in genomicsand proteomics. The book contains papers discussing some of the latest dataprocessing and visualization techniques and systems for effective analysis ofdiverse, large, complex, and multi-source data.

Internationally leading experts in the area of data visualization came to-gether for a workshop dedicated to visualization in medicine and life sciences,held on the island of Rugen, Germany, in July 2006. About 40 participantspresented state-of-the-art research on this topic. Research and survey paperswere solicited and carefully refereed, resulting in this collection.

The research topics covered by the papers in this book deal with thesethemes:

• Segmentation and Feature Detection• Surface Extraction• Volume Visualization• Graph and Network Visualization• Visual Data Exploration• Multivariate and Multidimensional Data Visualization• Large Data Visualization

The workshop was supported, in part, by the Deutsche Forschungsgemein-schaft (DFG).

Bremen, Germany Lars LinsenKaiserslautern, Germany Hans HagenDavis, California, U.S.A. Bernd Hamann

June 2007

Contents

Part I Surface Extraction Methods from Medical Imaging Data

Towards Automatic Generation of 3D Models of BiologicalObjects Based on Serial SectionsVincent Jasper Dercksen, Cornelia Bruß, Detlev Stalling, SabineGubatz, Udo Seiffert, and Hans-Christian Hege . . . . . . . . . . . . . . . . . . . . . . 3

A Topological Approach to Quantitation of RheumatoidArthritisHamish Carr, John Ryan, Maria Joyce, Oliver Fitzgerald, DouglasVeale, Robin Gibney, and Patrick Brennan . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3D Visualization of Vasculature: An OverviewBernhard Preim and Steffen Oeltze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3D Surface Reconstruction from Endoscopic VideosArie Kaufman and Jianning Wang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Part II Geometry Processing in Medical Applications

A Framework for the Visualization of Cross Sectional Datain Biomedical ResearchEnrico Kienel, Marek Vanco, Guido Brunnett, Thomas Kowalski,Roland Clauß, and Wolfgang Knabe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Towards a Virtual Echocardiographic Tutoring SystemGerd Reis, Bernd Lappe, Sascha Kohn, Christopher Weber,Martin Bertram, and Hans Hagen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Supporting Depth and Motion Perceptionin Medical Volume DataJennis Meyer-Spradow, Timo Ropinski, and Klaus Hinrichs . . . . . . . . . . . 121

VIII Contents

Part III Visualization of Multi-channel Medical Imaging Data

Multimodal Image Registration for Efficient Multi-resolutionVisualizationJoerg Meyer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

A User-friendly Tool for Semi-automated Segmentationand Surface Extraction from Color Volume DataUsing Geometric Feature-space OperationsTetyana Ivanovska and Lars Linsen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Part IV Vector and Tensor Visualization in Medical Applications

Global Illumination of White Matter Fibersfrom DT-MRI DataDavid C. Banks and Carl-Fredrik Westin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Direct Glyph-based Visualization of Diffusion MR DataUsing Deformed SpheresMartin Domin, Sonke Langner, Norbert Hosten, and Lars Linsen . . . . . . 185

Visual Analysis of Bioelectric FieldsXavier Tricoche, Rob MacLeod, and Chris R. Johnson . . . . . . . . . . . . . . . . 205

MRI-based Visualisation of Orbital Fat DeformationDuring Eye MotionCharl P. Botha, Thijs de Graaf, Sander Schutte, Ronald Root, PiotrWielopolski, Frans C.T. van der Helm, Huibert J. Simonsz, andFrits H. Post . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Part V Visualizing Molecular Structures

Visual Analysis of Biomolecular SurfacesVijay Natarajan, Patrice Koehl, Yusu Wang, and Bernd Hamann . . . . . . 237

BioBrowser – Visualization of and Access to Macro-MolecularStructuresLars Offen and Dieter Fellner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Visualization of Barrier Tree Sequences RevisitedChristian Heine, Gerik Scheuermann, Christoph Flamm,Ivo L. Hofacker, and Peter F. Stadler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

Contents IX

Part VI Visualizing Gene Expression Data

Interactive Visualization of Gene Regulatory Networkswith Associated Gene Expression Time Series DataMichel A. Westenberg, Sacha A. F. T. van Hijum, Andrzej T. Lulko,Oscar P. Kuipers, and Jos B. T. M. Roerdink . . . . . . . . . . . . . . . . . . . . . . . 293

Segmenting Gene Expression Patterns of Early-stageDrosophila EmbryosMin-Yu Huang, Oliver Rubel, Gunther H. Weber, Cris L. LuengoHendriks, Mark D. Biggin, Hans Hagen, and Bernd Hamann . . . . . . . . . . 313

Color Plates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Part I

Surface Extraction Methods from MedicalImaging Data

Towards Automatic Generation of 3D Modelsof Biological Objects Based on Serial Sections

Vincent Jasper Dercksen1, Cornelia Bruß2, Detlev Stalling3, SabineGubatz2, Udo Seiffert2, and Hans-Christian Hege1

1 Zuse Institute Berlin, Germany dercksen,hege@zib.de2 Leibniz Institute of Plant Genetics and Crop Plant Research, Gatersleben,

Germany bruess,gubatz,seiffert@ipk-gatersleben.de3 Mercury Computer Systems Inc., Berlin, Germany dstallin@mc.com

Summary. We present a set of coherent methods for the nearly automatic creationof 3D geometric models from large stacks of images of histological sections. Three-dimensional surface models facilitate the visual analysis of 3D anatomy. They alsoform a basis for standardized anatomical atlases that allow researchers to integrate,accumulate and associate heterogeneous experimental information, like functionalor gene-expression data, with spatial or even spatio-temporal reference. Models arecreated by performing the following steps: image stitching, slice alignment, elasticregistration, image segmentation and surface reconstruction. The proposed methodsare to a large extent automatic and robust against inevitably occurring imagingartifacts. The option of interactive control at most stages of the modeling processcomplements automatic methods.

Key words: Geometry reconstruction, surface representations, registration,segmentation, neural nets

1 Introduction

Three-dimensional models help scientists to gain a better understanding ofcomplex biomedical objects. Models provide fundamental assistance for theanalysis of anatomy, structure, function and development. They can for exam-ple support phenotyping studies [J.T06] to answer questions about the relationbetween genotype and phenotype. Another major aim is to establish anatom-ical atlases. Atlases enable researchers to integrate (spatial) data obtained bydifferent experiments on different individuals into one common framework. Amultitude of structural and functional properties can then jointly be visual-ized and analyzed, revealing new relationships. 4D atlases can provide insightinto temporal development and spatio-temporal relationships.

We intend to construct high-resolution 4D atlases of developing organisms,that allow the integration of experimental data, e.g. gene expression patterns.

4 V.J. Dercksen et al.

Such atlases can support the investigation of morphogenesis and gene expres-sion during development. Their creation requires population-based averagesof 3D anatomical models representing different developmental stages. By in-terpolating these averages the development over time can be visualized.

In this work we focus on the mostly automatic creation of individual 3Dmodels as an essential step towards a standard atlas. Individual models can becreated from stacks of 2D images of histological sections or from true 3D imagedata. With histological sections higher resolutions are possible than for exam-ple with 3D Nuclear Magnetic Resonance (NMR) imaging. Furthermore, fluo-rescent dye penetration problems, which can occur when imaging (thick) planttissue with for example (3D) Confocal Laser Scanning Microscopy (CLSM),can be avoided. During data acquisition often multiple images per section haveto be created to achieve the desired resolution. These sub-images have to bestitched in an initial mosaicing step. The 3D model construction then contin-ues with the alignment of the image stack to restore the 3D coherence. In thefollowing segmentation step, the structures of interest need to be identified,delimited and labeled. The model is completed by creating a polygonal sur-face, marking the object boundary and separating the structures it consistsof. This sequence of nontrivial processing steps is here called the geometryreconstruction pipeline (see Fig. 1).

With our flexible general-purpose 3D visualization system [SWH05], threehighly resolved 3D models of grains have recently been created basically man-ually [S. 07] (see Fig. 2(a)). Experience gained during the highly interactivemodeling however clearly showed the need for the automation and facilitationof the many repetitive, time-consuming, and work-intensive steps.

When creating such detailed models, the size of the data sets is a com-plicating factor. Due to the high resolution of the images and the size of thesubject, the data sets frequently do not fit into the main memory of commonworkstations and must therefore be processed out-of-core. Another problemis that due to the cutting and handling, histological sections are susceptibleto imaging artifacts, like cracks, contrast differences and pollution. Very ro-bust processing methods that are able to deal with such imperfect data aretherefore required.

Fig. 1. From physical grains to 3D models: processing steps of the geometry recon-struction pipeline.

Generation of 3D Models of Biological Objects 5

(a)

(b)

(c)

Fig. 2. (a) Example of a 3D surface model of a grain, created by a basically manualmodeling procedure [S. 07]. (b) Exemplary cross-section image (no. 160) assembledfrom 4 sub-images (nos. 640 - 643). (c) A regular grid of rectangular cutouts isused to find the correct position of this sub-image with respect to the previous one(not shown). The optimal position is searched for each cutout. Here three patcheshave been successfully matched with an adequate translation vector and sufficientcorrelation values (darkened squares), the result is considered reliable and the searchfor the remaining patterns is skipped. The medium gray cutouts are not consideredbecause of too little texture.

Early work for digitally representing and managing anatomical knowl-edge in graphical 3D models was concerned with human anatomy [K. 92,STA96, J.F99], later with model organisms, e.g. mouse [R. 03, M. 97, DRJ01]or fly [PH04]. Since the beginning, neuroscience has been a dominant appli-cation area, resulting in the generation of various brain atlases, e.g. [J.C97,RZH99, PH04, R. 05, A. 04, J.P05].

The problem of 3D model construction based on 2D section images has along history, going back to the 1970’s [LW72]. Though much work has beendone since then [SWM97, U. 05, Ju05], the proper alignment of the sectionsstill remains a difficult problem, especially when there is no 3D referenceobject available.

In plant biology, NMR imaging has been used to visualize developing bar-ley caryopses, including the distribution of chemical compounds [Gli06]. Directvolume rendering visualizations of rice kernels [Y. 01] and soybeans [KOS02]

6 V.J. Dercksen et al.

based on 2D serial sections have also been reported. A relatively new techniquefor 3D imaging is Optical Projection Tomography (OPT) [K. 06]. Haseloff[Has03] has created three-dimensional models of Arabidopsis meristem cellsbased on CLSM data. So far, no large scale 3D anatomical, geometrical mod-els of plant seeds, based on serial sections have been created using automaticmethods.

Here we propose a coherent set of methods to reconstruct 3D surface mod-els from high-resolution serial sections with minimal user interaction. This setconsists of: 1) a robust automatic stitching algorithm; 2) a fast automatic rigidalignment algorithm included in an interactive environment, which provideseffective visual feedback and allows for easy interactive correction-making;3) an elastic registration method, which compensates for non-linear deforma-tions; 4) automatic labelling achieved by an artificial neural network-basedsegmentation; and 5) an algorithm for the simultaneous triangulation andsimplification of non-manifold surfaces from out-of-core label data. All meth-ods are able to process out-of-core data and special attention was paid to therobustness of all automatic algorithms.

We applied these methods exemplarily for the reconstruction of plantseeds. However, all methods (except segmentation) can also be applied todata sets of different objects, and/or to data obtained with different imagingtechniques, e.g., different histological staining or true 3D data. In the lattercase one would skip the first part of the pipeline and start with the segmen-tation. To obtain a good segmentation of complex objects, it is imperative toincorporate expert knowledge. The presented segmentation method uses suchknowledge and is therefore problem-specific. The neural network approach ishowever generally applicable. Each part of the pipeline will be described indetail in the next section.

2 Geometry Reconstruction Pipeline

2.1 Image Pre-Processing

The input for the reconstruction pipeline is a set of sub-images, each repre-senting a part of a histological section (see Fig. 2(b)). In an automatic pre-processing step the sub-images are stitched to form a single image for eachsection. Szeliski [Sze05] provides an overview of image stitching. The set ofn sub-images constituting a particular sectional image is known in advance.Neighboring sub-images normally share a common region of overlap, whichforms the basis for the computation of their correct relative positions. For nsub-images there are n(n − 1)/2 possible combinations to check for overlap.In our case, we know that successively photographed images share a commonregion which reduces the number of pairs of sub-images to investigate to n−1plus one additional check of the last to the first sub-image for consistency andreliability.

Generation of 3D Models of Biological Objects 7

The translation vector TI2 = (xI2 , yI2) which optimally positions image I2

with respect to I1 is implicitly determined via the translation vector TP for asmall cutout P of I2. The latter is found by using the windowed NormalizedCross-Correlation (NCC) [GW02]:

NCC(x, y) =

I−1∑i=0

J−1∑j=0

(I1(i + x, j + y) − E(I1(x, y, I, J))

) (P (i, j) − E(P )

)std(I1(x, y, I, J)) std(P )

(1)where I and J mark the size of P , while E and std represent the expectationvalue and the standard deviation of the overlapping cutouts in both gray-value sub-images, respectively. The maximum of the NCC corresponds to thetranslation vector resulting in the best match between P and I1. To be con-sidered reliable, the maximum NCC value must exceed an empirically definedthreshold. The search process is accelerated by using a multi-scale approach.Using a cutout P instead of an entire image I2 ensures comparable correlationvalues, as the NCC is computed from a constant number (I ∗J) of values (theinsertion positions are chosen such that I1(i+x, j +y) is always defined). Thesize of P should be chosen such that it is smaller than the region of overlapand large enough to obtain statistically significant NCC values.

For reliability purposes, we compute the NCC for multiple cutouts of I2.An incorrect T , e.g. caused by a match of a cutout containing a movingparticle, is effectively excluded by searching for an agreeing set of patches/translation vectors (see Fig. 2(c)). In rare cases an image cannot be reliablyassembled. Such cases are automatically detected and reported. Appropriateuser actions – such as an additional calculation trial of a translation vectorwith a manually defined cutout or a visual check in case of arguable correlationvalues – are provided.

2.2 Alignment

In the alignment step, the three-dimensional coherence of the images in thestack, lost by sectioning the object, is restored. The input for this step is anordered stack of stitched section images. This step is divided into two regis-tration steps: a rigid alignment restores the global position and orientation ofeach slice; the subsequent elastic registration step compensates for any non-linear deformations caused by, e.g., cutting and handling of the histologicalsections. The latter is required to obtain a better match between neighbor-ing slices, resulting in a smoother final surface model. Maintz [MV98] andZitova [ZF03] provide surveys of the image registration problem.

The rigid alignment of the image stack is approached as a series of pair-wise 2D registration problems, which can be described as follows: for each sliceR(ν) : Ω ⊂ R

2 → R, ν ∈ 2, . . . , M, find a transformation ϕ(ν), resulting ina slice R′(ν) = R(ν) ϕ(ν), such that each pair of transformed consecutiveslices (R′(ν−1), R′(ν)) matches best. Allowed are translations, rotations and