simplifies the load-balancing, datamovement, unstructured-communica-tion, and memory usage difficulties thatarise in dynamic applications such asadaptive finite-element methods, parti-cle methods, and crash simulations.Zoltan’s data-structure-neutral designalso lets a wide range of applications useit without imposing restrictions on ap-plication data structures. Its object-based interface provides a simple andinexpensive way for application devel-opers to use the library and researchersto make new capabilities available undera common interface.

Zoltan provides tools that help appli-cation developers without imposingstrict requirements on them. For exam-ple, it includes a suite of parallel parti-tioning algorithms and data migrationtools that redistribute data to reflect,say, changing processor workloads.Zoltan also includes distributed data di-rectories and unstructured communica-tion services that let applications per-form complicated communication usingonly a few simple primitives. To simplifydebugging of dynamic memory usage,Zoltan provides dynamic memory man-agement tools that enhance commonmemory allocation functions. In this ar-ticle, we describe Zoltan’s features andways to use it in dynamic applications.

Zoltan software designOur design of the Zoltan library does

not restrict it to any particular type ofapplication. Rather, Zoltan operates onuniquely identifiable data items that wecall objects. For example, in finite-ele-ment applications, objects might be el-ements or nodes of the mesh. In parti-cle applications, objects might beparticles. In linear solvers, objectsmight be matrix rows. Each objectmust have a unique global identifier(ID) represented as an array of un-signed integers. Common choices in-clude global numbers for elements(nodes, rows, particles, and so on) thatalready exist in many applications, or astructure consisting of an owningprocessor number and the object’s lo-cal-memory index. Objects might alsohave local (to a processor) IDs that donot have to be unique globally. LocalIDs such as addresses or local-array in-dices of objects can improve the per-formance (and convenience) of Zoltan’sinterface to applications.

To make Zoltan easy to use, we donot impose any particular data struc-ture on an application, nor do we re-quire an application to build a particu-lar data structure for Zoltan. Instead,Zoltan uses a callback function inter-face in which the tool queries the ap-

plication for needed data. The applica-tion must provide simple functionsthat answer these queries.

For example, Figure 1 shows howZoltan’s callback function interfaceworks for performing dynamic load bal-ancing in an application. An applicationstarts Zoltan (Zoltan_Initialize)and allocates the memory it needs(Zoltan_Create). Through calls toZoltan_Set_Fn, the application passespointers to its callback functions toZoltan. It also selects a partitioningmethod (Zoltan_LB_Set_Method)and sets appropriate parameters for loadbalancing(Zoltan_LB_Set_Param).Then, within the main computationloop, the application calls Zoltan_LB_Balance to compute a new partition ofits data.

As a first step in Zoltan_LB_Bal-ance, Zoltan must build the data struc-tures needed for the particular parti-tioning method selected. It calls the

90 1521-9615/02/$17.00 © 2002 COMPUTING IN SCIENCE & ENGINEERING

ZOLTAN DATA MANAGEMENT SERVICESFOR PARALLEL DYNAMIC APPLICATIONSBy Karen Devine, Erik Boman, Robert Heaphy, Bruce Hendrickson, and Courtenay Vaughan

THE ZOLTAN LIBRARY IS A COLLECTION OF DATA MANAGE-

MENT SERVICES FOR PARALLEL, UNSTRUCTURED, ADAP-

TIVE, AND DYNAMIC APPLICATIONS THAT IS AVAILABLE AS OPEN-

SOURCE SOFTWARE FROM WWW.CS.SANDIA.GOV/ZOLTAN. IT

Editor: Paul F. Dubois, paul@pfdubois.com

PROGRAMMINGS C I E N T I F I C P R O G R A M M I N G

Zoltan Details

The Zoltan library’s toolkit includesparallel partitioning algorithms, datamigration tools, distributed directo-ries, and unstructured communica-tion and memory managementpackages. It has many key featuresfor developers:

• Its source code is freely available atwww.cs.sandia.gov/Zoltan.

• It’s callable from C, C++, and F90(but is implemented in C).

• It uses MPI communication.

MARCH/APRIL 2002 91

Café Dubois

XP Torture TestBoth my kids’ computers had a Windows XP upgrade

coming to them. My daughter is at college where I can’thelp her when things break, and my son is an operating sys-tem programmer’s worst nightmare. Therefore I decided,despite my opposition to the XP “activation” requirement,to do the upgrades because of the alleged improvement inreliability. There aren’t many areas of life like this one, wherewe buy another product from someone because the lastone they sold us is so defective.

My daughter brought her Vaio home for Christmas. Sony’supgrade included a second CD that prepared the machine,instructed you to do the upgrade, and then returned to re-place even more of your software. They did a good job and itall worked, but it took about two hours. For some reason, itdid not install the network configuration correctly, somehowdisabling Dubois Castlenet and angering an impatient mob.

In my son’s case, he had worked Windows Me into an in-teresting state where every attempt to open a foldercrashed Explorer. Figuring that the upgrade was not likely towork very well in such an environment, I reformatted hisdrive and reinstalled from his original Windows 98-2e OEMdisks. After the usual search team was sent out to find theproduct key, I was able to start the 98-2e -> XP upgrade. Anadditional 90 minutes of sheer boredom punctuated by mo-ments of terror later, I had a working machine with nosound. Another two hours were spent figuring that out.

Both machines later announced that the DVD decoder was toastand that I should click here to go get a new one. My son’s manufac-turer charged me $30 for it. I was too tired to sue.

A friend who went through upgrading on his machine saidhe only got “a few heart-stopping blue screens of death” dur-ing the process.

Cavaet upgraditor. Carpe monopolist.

BuildingI spend a lot of time at work wrestling with our build. Our

product is an open-source, Python-based set of tools for sci-entific analysis and visualization with a special emphasis onclimate data (cdat.sf.net). As such it spans many directoriesand also requires the building software we need such asPython, zlib, Tcl, Tk, and readline on a variety of machines.Each of these pieces has its own configuration procedures.

Until Python got its “Distutils” that help you build Pythonpackages across platforms, things were in an untenablemess. But now we can build our packages pretty well withjust a few frightening problems. Most of these problems fallinto one of these categories:

• Headers and libraries from the software we use, such asX11, do not have standardized locations.

• Some OSs include different things than others.

• Some OSs come with versions of things that we use butthey are older versions that we cannot use and must some-how avoid.

• Command lines for compilers and linkers are not standardized.• Make is not standardized.• Many users don’t read the README before installing.• Cross-platform portability to Windows and Mac seems even

farther out of reach.

Here are some resources for dealing with this problem:• Autoconf (www.gnu.org/manual/autoconf ) is widely used

but is hard to learn and use; too hard for scientists, I think.Am I wrong?

• Imake (www.dubois.ws/software/imake-stuff) was some-thing my group tried for a year or so but again it was toohard for scientists. (This Web site by Paul Dubois is not myWeb site. As Yoda said, there is another.)

• Cons (www.dsmit.com/cons) is a Perl-based system thatsome people think highly of.

• SCons (www.scons.org) is an implementation of the win-ning design for a build tool in the Software Carpentry con-test. It is just getting going. I think it has a lot of potentialbecause it lets you write the configuration file (the equiva-lent of a Makefile) in Python. SCons is supposed to “fixwhat is wrong with Cons.” I tried SCons and it built a Ccode on Windows without a problem. That is usually a ter-ror-filled process without building a Visual Studio project.

• JAM / MR (perforce.com/jam/jam.html) is another systemthat has attracted an enthusiastic audience. I haven’t hadtime to look into it, but the same people wrote Perforce, myfavorite source-code control system, so I know they live inthe real world of having a complex multiplatform product.

Send me your tales of woe and success and maybe we cantogether figure out what works. Meantime, I’ll time how longit takes my son to destroy XP.

92 COMPUTING IN SCIENCE & ENGINEERING

registered callback functions to build itsdata structures. It then performs the par-titioning and returns lists of objects to beimported to and exported from theprocessor for the new decomposition. Atno time does the application developerhave to build (or debug) complicateddata structures for use within Zoltan.

To keep the application interfacesimple, we use a small set of callbackfunctions and make them easy to writeby requesting only information that iseasily accessible to applications. Forthe most basic partitioning algorithms,Zoltan requires only four callbackfunctions. These functions return thenumber of objects owned by a proces-sor, a list of weights and IDs for ownedobjects, the problem’s dimensionality,and a given object’s coordinates. (Fig-ure 2 includes simple examples of someof these callback functions.) More so-phisticated graph-based partitioning

algorithms require only two additionalcallback functions, which return thenumber of edges per object and edgelists for objects.

Zoltan’s parallel data servicesZoltan provides a toolkit of parallel

data services for dynamic and unstruc-tured applications. These services arelayered in Zoltan so that applicationdevelopers can use as much or as littleof the toolkit as desired. Zoltan’s paral-lel partitioning tools are efficient algo-rithms for dividing an application’s dataamong processors while trying to min-imize interprocessor communication.Incremental partitioning algorithms—algorithms that account for data’s cur-rent location in determining their newlocation—are included to keep datamovement costs low. Zoltan also in-cludes data migration utilities, distrib-uted data directories, an unstructured

communication package, and a dy-namic memory management package.

Parallel partitioning anddynamic load balancing

In our experience, no single parti-tioning strategy is effective for all paral-lel computations. Some applications re-quire partitions based only on theproblem’s workloads and geometry;others benefit from explicit considera-tion of dependencies between objects.Some applications require the highest-quality partitions possible, regardless ofthe cost to generate them; others cansacrifice some quality so long as newpartitions can be generated quickly. Forsome applications, the cost to relocatedata is prohibitively high, so incremen-tal partitioning algorithms are needed;other applications can tolerate greaterremapping costs. Most important, ap-plication developers might not know inadvance which strategies work best intheir applications, so they need a conve-nient means of comparing algorithms.

We provide three classes of parallelpartitioning algorithms in the Zoltan li-brary: geometric bisection, space-fillingcurves, and graph partitioning. Eachclass includes several different algo-rithms. Once users write the callbackfunctions for each class, switching be-tween classes and methods requires onlya call to Zoltan_LB_Set_Method withthe new algorithm name. In this way,developers can easily compare algo-rithms within their applications to findthe strategy that works best for them.

Geometric bisection. Recursive coordi-nate bisection is conceptually the sim-plest partitioning algorithm in Zoltan.1

RCB divides a problem’s work into twoequal parts using a cutting plane or-thogonal to a coordinate axis and assignsobjects to subdomains based on theirgeometric positions relative to the cut-ting plane (see Figure 3). It then recur-

S C I E N T I F I C P R O G R A M M I N G

Initialize Zoltan(Zoltan_Initialize,

Zoltan_Create)

Select partitioning method(Zoltan_LB_Set_Method,

Zoltan_Set_Params)

Register callback functions(Zoltan_Set_Fn)

Repartition(Zoltan_LB_Balance)

Compute

Move data(Zoltan_Help_Migrate)

Clean up (Zoltan_Destroy)

Application

Zoltan_LB_Balance:

• Call registered callback function

• Build data structures for method• Compute new decomposition

• Return import/export lists

Zoltan_Help_Migrate:

• Call packing callback function for exports• Send exports• Receive imports• Call unpacking callback function for imports

Zoltan

Figure 1. An example shows how to use Zoltan’s callback function interface for parallel partitioning in applications. The application (left) initializes Zoltan, selects apartitioning method, and passes pointers to its callback functions to Zoltan. Whenparallel partitioning is invoked, Zoltan (right) uses the callback functions to build itsdata structures to perform partitioning. A similar callback function interface helpsmigrate data between processors after a new partition is computed.

MARCH/APRIL 2002 93

sively cuts the resulting subdomains un-til the number of subdomains is equal tothe number of processors. Recursive in-ertial bisection (RIB) is a variant of RCBthat chooses cutting planes orthogonalto the principle axes of the geometry,rather than to the coordinate axes.2,3

RCB is effective for many applica-tions because it maintains geometric lo-cality of objects in processors; it is alsofast and implicitly incremental (smallchanges in workloads produce onlysmall changes in the decomposition). Itis widely used in crash simulations andparticle simulations because it assignsphysically close surfaces or particles toa single processor. Figure 4 shows anadaptive mesh-refinement applicationthat uses RCB to repartition after localrefinement. RCB assigns parent andchild elements to the same processor,preventing the need for communicationbetween mesh levels.

Space-filling curves (SFC). Researchershave used space-filling curves (inventedin the 1890s by Peano and Hilbert) forpartitioning in gravitational simulations,smoothed-particle hydrodynamics, andadaptive finite-element methods. LikeRCB, SFC partitioners rely only on theweights and geometric coordinates of ob-jects. Figure 5 shows how the SFC algo-rithm works. The SFC partitioner refinesthe computational domain in Figure 5ainto subregions of a desired granularity(for example, one subregion per object).A subregion’s weight is the sum of theworkloads of all objects in that subregion.The partitioner then passes an SFCthrough the subregions, mapping a mul-tidimensional domain to a one-dimen-sional line (see Figure 5b). The parti-tioner cuts the line into equally weightedsegments and assigns all objects along asegment to a single processor (see Figure5c). Like RCB, SFC partitioning is fastand incremental and assigns geometri-cally close objects to a single processor.

Zoltan includes several variants ofSFC partitioning. Octree partitioningexplicitly builds the tree representingthe refinement of a domain into subre-gions; traversals of the tree produceSFCs.4 Binned SFC partitioning effec-tively lets several objects exist in a sin-gle subregion; adaptive refinement of

the domain efficiently obtains the de-sired granularity (see the user’s guide atwww.cs.sandia.gov/Zoltan). Refine-ment tree partitioning is designedspecifically for adaptive mesh-refine-ment applications.5 It uses parent–childrelationships rather than geometric co-ordinates to construct a tree that re-

int Number_Owned_Nodes;struct Node_Type {

double Coordinates[3]; float Weight; int Global_ID_Num; } Nodes[MAX_NODES];

void owned_objects_callback(void *data, int num_gid_entries, int num_lid_entries; ZOLTAN_ID_PTR global_ids, ZOLTAN_ID_PTR local_ids, int wgt_dim, float *obj_wgts, int *ierr){int i; /* return global node numbers as global_ids. */ /* return index into Nodes array for local_ids. */ for (i = 0; i < Number_Owned_Nodes; i++)[ global_ids[i] = Nodes[i].Global_ID_Num; local_ids[i] = i; obj_wgts[i] = Nodes[i].Weight; } *ierr = ZOLTAN_OK;}

void coordinates_callback(void *data, int num_gid_entries, int num_lid_entries, ZOLTAN_ID_PTR global_id, ZOLTAN_ID_PTR local_id, double *geom_vec, int *ierr){ /* use local_id to index into the Nodes array. */ geom_vec[0] = Nodes[local_id[0]].Coordinates[0]; geom_vec[1] = Nodes[local_id[0]].Coordinates[1]; geom_vec[2] = Nodes[local_id[0]].Coordinates[2]; *ierr = ZOLTAN_OK;}

Figure 2. Example callback functions for a simple mesh-based application. Finite-element nodes are the objects to be passed to Zoltan. The function owned_objects_callback returns a list of node weights and IDs for each node owned by a processor.The function coordinates_callback returns a given node’s coordinates.

3rd

3rd

3rd

3rd

2nd

2nd

1st cut Figure 3. A diagram of cuts madeduring recursive coordinate bisection partitioning (RCB).Recursive cuts orthogonal tothe coordinate axes divide adomain’s work into evenlysized subdomains.

94 COMPUTING IN SCIENCE & ENGINEERING

flects the domain’s refinement; traver-sals of this tree also produce SFCs.

Graph partitioning. In graph parti-tioning algorithms, the problem to bepartitioned is represented as a weightedgraph. The graph’s weighted nodes rep-resent the objects to be partitioned (seeFigure 6). Graph edges represent de-pendencies between objects and serveas an approximation of communicationcosts. Graph partitioning algorithmsdivide the graph to assign roughly equalnodal weight to partitions while at-tempting to minimize the weight ofedges cut by partition boundaries.

Zoltan provides graph partitioning

through interfaces to the popular graph-partitioning packages ParMETIS andJostle.6,7 Both ParMETIS and Jostleprovide parallel implementations of themultilevel graph-partitioning approachpioneered in Chaco.8,9 Multilevel algo-rithms first coarsen an input graph, withcoarse nodes representing clusters of in-put-graph nodes. The coarse graph iseasy to partition. The multilevel algo-rithms then project this coarse partitionback to the input graph, with local im-provements of the partition performedat each finer graph level. This approachproduces high-quality partitions formany applications (see Figure 7). How-ever, multilevel graph partitioning ismore expensive than the geometricmethods described earlier, and the par-titions produced are not incremental.

ParMETIS and Jostle also include dif-fusive graph-partitioning algorithms.10

Diffusive algorithms move work fromheavily loaded processors to more lightlyloaded neighboring ones. The problem’sgraph helps determine which objectsshould be moved to which neighbors.Diffusive algorithms generate partitionswhose quality can degrade over several

invocations of the partitioning algo-rithm. However, they are fast and incre-mental and can be effective for severaldynamic applications.

Data migration toolsA complicated part of dynamic repar-

titioning is the need to move data fromold processors to new ones. This datamigration requires deletions and inser-tions from the application data struc-tures as well as communication be-tween the processors. A general-purpose library such as Zoltan can dolittle to help manipulate applicationdata structures, but it can help commu-nicate object data among processors.

To help an application with data mi-gration, Zoltan requires an applicationto supply a callback function that packsa given object’s data into a communi-cation buffer. A function that unpacksan object’s data from a communicationbuffer must also be supplied by the ap-plication. Zoltan uses these callbackfunctions just as it used callback func-tions for repartitioning in Figure 1. Foreach object to be exported, Zoltan callsthe packing function to load commu-

S C I E N T I F I C P R O G R A M M I N G

Figure 4. An RCB decomposition for anadaptive mesh-refinement application.Colors indicate processor assignments.(Image courtesy of Carter Edwards andhis colleagues at Sandia National Laboratories.)

9

20

19

18

17

16

15

14

1312

1110

8

7

6 5

4

321

9

20

19

18

17

16

15

14

1312

1110

8

7

6 5

4

321

9

20

19

18

17

16

15

14

1312

1110

8

7

65

4

321

(b) (c)(a)

Figure 5. Space-filling curve partitioning for four processors: (a) the original domain with particles to be partitioned; (b) the refined domain with space-filling curve (red) and linear ordering of particles; and (c) a partition of the linear ordering to assignan equal number of particles to processors.

MARCH/APRIL 2002 95

nication buffers, performs all commu-nication needed to move the data, andthen calls the unpacking function toload that object’s data into the datastructures on its new processor.

Distributed data directoriesDynamic applications often need to

locate off-processor information. Afterrepartitioning, a processor might needto rebuild ghost cells and lists of objectsto be communicated. It might knowwhich objects it needs but not wherethey are. To help locate off-processordata, Zoltan includes a distributed datadirectory algorithm based on Ali Pinar’srendezvous algorithm.11 Figure 8 showshow to use the directory. Processorsregister their owned objects’ IDs alongwith their processor number in a direc-tory (by calling Zoltan_DD_Update).This directory is distributed evenlyacross processors predictably (througheither a linear decomposition of the IDsor a hashing of IDs to processors).Then, other processors can obtain agiven object’s processor number bysending a request for the information tothe processor holding the directory en-try (by calling Zoltan_DD_Find).

Unstructured communicationlibrary

Unlike static applications wherecommunication patterns remain fixedthroughout the computation, dynamicapplications can have complicated,changing communication patterns. Forexample, after adaptive mesh refine-ment, new communication patternsmust reflect dependencies betweennewly created elements. Multiphysicssimulations, such as crash simulations,might require complicated communi-cation patterns to transfer data be-

tween decompositions for differentsimulation phases.

Zoltan provides an unstructuredcommunication package to simplifycommunication. It generates a com-munication plan based on the numberof objects to be sent and their destina-tion processors. This plan includes in-formation about both the sends and re-ceives for a given processor. The plancan be used and reused throughout theapplication or destroyed and rebuiltwhen communication patterns change.It can also be used in reverse to return

Figure 6. A representation of a triangular finite-element mesh as agraph (red) for graph partitioning.Graph nodes represent mesh elements;graph edges represent shared elementfaces.

Figure 7. A multilevel graph-based partition of a finite-element mesh for six processors. Colors represent processor assignments of finite-element nodes. (Mesh courtesy of Steve Hammond, NCAR.)

Allocate memory for distributed directory(Zoltan_DD_Create)

Register owned objects and/or updateprocessor assignments of migrated objects

(Zoltan_DD_Update)

Find processor assignments of needed objects(Zoltan_DD_Find)

Free memory used by distributed directory(Zoltan_DD_Destroy)

Figure 8. An outline of distributed data directoryusage. Once created, directories can be updatedand used throughout anapplication to reflect, say,changing processor assignments due to dynamic load balancing.

96 COMPUTING IN SCIENCE & ENGINEERING

data to requesting processors. Thepackage includes simple communica-tion primitives that insulate the userfrom details of sends and receives.

Figure 9 shows how to use the com-munication package to transfer databetween two different meshes in aloosely coupled physics simulation.This crash simulation uses a staticgraph-based decomposition generatedby Chaco (left) for the finite-elementanalysis and a dynamic RCB decompo-sition (right) for contact detection.Through a call to Zoltan_Comm_Cre-ate, the application obtains a commu-nication plan (built by Zoltan) that de-scribes data movement between thetwo decompositions. Using the plan,the application can transfer data be-tween the graph-based and RCB de-compositions through calls toZoltan_Comm_Do and Zoltan_Comm_Do_Reverse.

Dynamic memory managementpackage

Dynamic applications rely heavily onthe ability to allocate and free memoryas needed. After repartitioning, for ex-ample, new memory is needed for im-

ported objects, and exported objects’memory is freed. Memory leaks andinvalid memory accesses are commonin developing software. Although manysoftware development tools let userstrack memory bugs, these tools are of-ten not available on state-of-the-artparallel-computing platforms. Thus,simple in-application debugging toolscan be beneficial.

Zoltan’s memory management pack-age includes wrappers around mallocand free that provide enhanced de-bugging capability for memory man-agement. The arguments to thesewrappers are identical to those passedto malloc and free, so they are easy touse in applications. For each malloc,the memory management packagerecords the line number and functionname of where the allocation tookplace. When a location is freed, its al-location record is removed. Thus,tracking memory leaks can be simpli-fied by asking the memory manage-ment package to print its list of un-freed memory along with thesource-code location from which itwas allocated. Statistics about memoryallocations and frees are also available.

A lthough we designed Zoltan to bea general-purpose tool for appli-

cations, it has proven valuable to us asa research testbed for new software de-velopment. By allowing easy compari-son of partitioning algorithms, Zoltanis ideal for implementing and testingnew load-balancing strategies such asmulticonstraint geometric partitioningstrategies. Its flexible design lets us eas-ily add new technologies (such asgraph-coloring and graph-matching al-gorithms) to Zoltan. We are building alayer within Zoltan to provide parti-tioning and data services for heteroge-neous computing architectures.

AcknowledgmentsWe thank Steve Attaway, Carter Ed-

wards, Robert Leland, Ali Pinar, StevePlimpton, Alex Pothan, Robert Preis,and John Shadid for their discussions,ideas, and color figures.

References1. M. Berger and S. Bokhari, “A Partitioning

Strategy for Nonuniform Problems on Multi-processors,” IEEE Trans. Computers, vol. C-36,1989, pp. 279–301.

S C I E N T I F I C P R O G R A M M I N G

Figure 9. A demonstration of Zoltan’s unstructured communication package for loosely coupled physics. In this example, communicationprimitives simplify data mapping between two different decompositions. The left side shows graph-based decomposition; the right,RCB decomposition. (Image courtesy of Steve Attaway and his colleagues at Sandia National Laboratories.)

Zoltan_Comm_DoZoltan_Comm_Do

Zoltan_Comm_Do_ReverseZoltan_Comm_Do_Reverse

2. H. Simon, “Partitioning of UnstructuredProblems for Parallel Processing,” ComputingSystems in Eng., vol. 2, nos. 2–3, 1991, pp.135–148.

3. V. Taylor and B. Nour-Omid, “A Study of theFactorization Fill-in for a Parallel Implementa-tion of the Finite Element Method,” Int’l J.Numerical Methods Eng., vol. 37, 1994, pp.3809–3823.

4. R. Loy, Adaptive Local Refinement with OctreeLoad-Balancing for the Parallel Solution ofThree-Dimensional Conservation Laws, doctor-al dissertation, Dept. of Computer Science,Rensselaer Polytechnic Inst., 1998.

5. W. Mitchell, “A Comparison of Three FastRepartition Methods for Adaptive Grids,” Proc.9th SIAM Conf. Parallel Processing for ScientificComputing, SIAM, Philadelphia, 1999.

6. G. Karypis, K. Schloegel, and V. Kumar,ParMETIS: Parallel Graph Partitioning andSparse Matrix Ordering Library, tech. report97-060, Dept. of Computer Science, Univ. ofMinnesota, Minneapolis, 1997.

7. C. Walshaw, M. Cross, and M. Everett,“Parallel Dynamic Graph Partitioning forAdaptive Unstructured Meshes,” J. Paralleland Distributed Computing, vol. 47, no. 2, 15Dec. 1997, pp. 102–108.

8. B. Hendrickson and R. Leland, The ChacoUser’s Guide, version 2.0, tech. report SAND94-2692, Sandia Nat’l Laboratories,Albuquerque, N.M., 1994.

9. B. Hendrickson and R. Leland, “A Multilevel Al-gorithm for Partitioning Graphs,” Proc. Super-computing ’95, ACM Press, New York, 1995.

10. G. Cybenko, “Dynamic Load Balancing forDistributed Memory Multiprocessors,” J.Parallel and Distributed Computing, vol. 7,no. 2, 1989, pp. 279–301.

11. A. Pinar, Combinatorial Algorithms in ScientificComputing, doctoral dissertation, Dept. ofComputer Science, Univ. of Illinois, Urbana-Champaign, 2001.

Karen Devine is the principal investigator for the Zoltan project. Her interests include parallel

partitioning, adaptive numerical methods, and software development. She received her PhD in

computer science from Rensselaer Polytechnic Institute. Contact her at the Computation,

Computers, and Mathematics Center, Sandia Nat’l Labs, Albuquerque, NM 87185-1111;

kddevin@sandia.gov.

Erik Boman has a PhD in scientific computing and computational mathematics from Stanford

University. His technical interests include scientific computing, numerical linear algebra, and

combinatorial algorithms. Contact him at the Computation, Computers, and Mathematics

Center, Sandia Nat’l Labs, Albuquerque, NM 87185-1111; egboman@sandia.gov.

Robert Heaphy has a PhD in physics from the University of New Mexico. His research interests

include the development of adaptive real-time control systems and software development for

massively parallel systems. Contact him at the Computation, Computers, and Mathematics

Center, Sandia Nat’l Labs, Albuquerque, NM 87185-1111; rheaphy@sandia.gov.

Bruce Hendrickson received a PhD in computer science from Cornell University. His research

interests include algorithms and software tools for high-performance scientific computing. Contact

him at the Computation, Computers, and Mathematics Center, Sandia Nat’l Labs, Albuquerque,

NM 87185-1111; bahendr@sandia.gov.

Courtenay Vaughan received his PhD in applied mathematics from the University of Virginia. His

research focus is parallel computing. Contact him at the Computation, Computers, and

Mathematics Center, Sandia Nat’l Labs, Albuquerque, NM 87185-1111; ctvaugh@sandia.gov.

Submissions: Send two copies, one word-processed file and one PostScript file, of articles and proposals to Francis Sullivan, Editor in Chief, CiSE, 10662 LosVaqueros Circle, Los Alamitos, CA 90720-1314; cise@computer.org. Submissions should not exceed 6,000 words and 10 references. All submissions are subjectto editing for clarity, style, and space.

Editorial: Unless otherwise stated, bylined articles and departments, as well as product and service descriptions, reflect the author’s or firm’s opinion.Inclusion in CiSE does not necessarily constitute endorsement by the IEEE, the AIP, or the IEEE Computer Society.

Circulation: Computing in Science & Engineering (ISSN 1521-9615) is published bimonthly by the AIP and the IEEE Computer Society. IEEE Headquarters,Three Park Ave., 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, PO Box 3014, Los Ala-mitos, CA 90720-1314, phone +1 714 821 8380; IEEE Computer Society Headquarters, 1730 Massachusetts Ave. NW, Washington, DC 20036-1903; AIPCirculation and Fulfillment Department, 1NO1, 2 Huntington Quadrangle, Melville, NY 11747-4502. Annual subscription rates for 2001: $40 for Com-puter Society members (print only) and $52 for AIP member society members (print plus online). For more information on other subscription prices, seehttp://computer.org/subscribe or http://ojps.aip.org/cise/subscrib.html. Back issues cost $10 for members, $20 for nonmembers. This magazine isavailable on microfiche.

Postmaster: Send undelivered copies and address changes to Circulation Dept., Computing in Science & Engineering, PO Box 3014, Los Alamitos, CA90720-1314. Periodicals postage paid at New York, NY, and at additional mailing offices. Canadian GST #125634188. Canada Post Publications MailAgreement Number 0605298. Printed in the USA.

Copyright & reprint permission: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of UScopyright law for private use of patrons those articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code ispaid through the Copyright Clearance Center, 222 Rosewood Dr., Danvers, MA 01923. For other copying, reprint, or republication permission, write toCopyright and Permissions Dept., IEEE Publications Administration, 445 Hoes Ln., PO Box 1331, Piscataway, NJ 08855-1331. Copyright © 2002 by theInstitute of Electrical and Electronics Engineers Inc. All rights reserved.