Adaptive computing on the grid using AppLeS - Parallel and ... · 2APPLES: PRINCIPLES AND FUNDAMENTAL CONCEPTS Initiated in 1996 [2], the goals of the AppLeS project have been twofold

Adaptive Computing on the Grid Using AppLeSFrancine Berman, Member, IEEE Computer Society, Richard Wolski, Member, IEEE,

Henri Casanova, Member, IEEE, Walfredo Cirne, Holly Dail, Member, IEEE Computer Society,

Marcio Faerman, Student Member, IEEE, Silvia Figueira, Jim Hayes, Graziano Obertelli,

Jennifer Schopf, Member, IEEE, Gary Shao, Shava Smallen, Neil Spring,

Alan Su, and Dmitrii Zagorodnov

Abstract—Ensembles of distributed, heterogeneous resources, also known as Computational Grids, have emerged as critical

platforms for high-performance and resource-intensive applications. Such platforms provide the potential for applications to aggregate

enormous bandwidth, computational power, memory, secondary storage, and other resources during a single execution. However,

achieving this performance potential in dynamic, heterogeneous environments is challenging. Recent experience with distributed

applications indicates that adaptivity is fundamental to achieving application performance in dynamic grid environments. The AppLeS

(Application Level Scheduling) project provides a methodology, application software, and software environments for adaptively

scheduling and deploying applications in heterogeneous, multiuser grid environments. In this article, we discuss the AppLeS project

and outline our findings.

Index Terms—Scheduling, parallel and distributed computing, heterogeneous computing, grid computing.

æ

1 INTRODUCTION

A Computational Grid [1], or Grid, is a collection ofresources (computational devices, networks, online

instruments, storage archives, etc.) that can be used as anensemble. Grids provide an enormous potential of capabil-ities that can be brought to bear on large distributedapplications and are becoming prevalent platforms forhigh-performance and resource-intensive applications.

The development and deployment of applicationswhich can realize a Grid’s performance potential facetwo substantial obstacles. First, Grids are typicallycomposed from collections of heterogeneous resourcescapable of different levels of performance. Second, theperformance that can be delivered varies dynamically asusers with competing goals share resources, resources fail,

are upgraded, etc. Consequently, Grid applications mustbe able to exploit the heterogeneous capabilities of theresources they have at their disposal while mitigating anynegative effects brought about by performance fluctuationin the resources they use.

In this article, we describe the AppLeS project. AppLeS(which is a contraction of Application Level Scheduling) isa methodology for adaptive application scheduling. Wediscuss various examples of adaptively scheduled Gridapplications that use AppLeS and the performance they canachieve. We also describe software environments fordeveloping and/or deploying applications in a way thatleverages AppLeS methodology. Taken together, theseresults define a novel approach to building adaptive,high-performance, distributed applications for new distrib-uted computing platforms.

2 APPLES: PRINCIPLES AND FUNDAMENTAL

CONCEPTS

Initiated in 1996 [2], the goals of the AppLeS project havebeen twofold. The first goal has been to investigate adaptivescheduling for Grid computing. The second goal has been toapply research results to applications for validating theefficacy of our approach and, ultimately, extracting Gridperformance for the end-user. We have achieved these goalsvia an approach that incorporates static and dynamicresource information, performance predictions, applicationand user-specific information, and scheduling techniquesthat adapt to application execution “on-the-fly.” Based onthe AppLeS methodology, we have developed template-based Grid software development and execution systemsfor collections of structurally similar classes of applications(discussed in later sections of this article).

Although the implementation details differ for indivi-dual examples of AppLeS-enabled applications, all arescheduled adaptively and all share a common architecture.Each application is fitted with a customized schedulingagent that monitors available resource performance andgenerates, dynamically, a schedule for the application.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 14, NO. 4, APRIL 2003 369

. F. Berman and S. Smallen are with the San Diego Supercomputer Center,9500 Gilman Drive MC 0505, La Jolla, CA 92093-0505.E-mail: {berman, smallen}@sdsc.edu.

. R. Wolski and G. Obertelli are with the Department of Computer Science,University of California Santa Barbara, Santa Barbara, CA 93106-5110.E-mail: {rich, graziano}@cs.ucsb.edu.

. H. Casanova, H. Dail, M. Faerman, J. Hayes, G. Shao, A. Su, and D.Zagorodnov are with the Department of Computer Science and Engineer-ing, University of California San Diego, La Jolla, CA 92093-0114.E-mail: {casanova, hdail, mfaerman, jhayes, gshao, alsu, dzagorod}@sdsc.edu.

. W. Cirne is with the Departamento de Sistemas e Computacao, Av. AprıgioVeloso, s/n, Caixa Postal: 10.106, 58.109-970, Campina Grande, PB,Brazil. E-mail: [email protected].

. S. Figueira is with the Department of Computer Engineering, Santa ClaraUniversity, Santa Clara, CA 95053. E-mail: [email protected].

. J. Schopf is with the Distributed Systems Laboratory, Mathematics andComputer Science Division, Argonne National Laboratory, Building 221,9700 South Cass Avenue, Argonne, IL 60439-4844.E-mail: [email protected].

. N. Spring is with Computer Science and Engineering, Sieg Hall,University of Washington, Box 352350, Seattle, WA 98195-2350.E-mail: [email protected].

Manuscript received 31 Oct. 2001; revised 23 Oct. 2002; accepted 31 Oct.2002.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 115296.

1045-9219/03/$17.00 ß 2003 IEEE Published by the IEEE Computer Society

The individual steps followed by an AppLeS agent aredepicted in Fig. 1 and are detailed below.

1. Resource Discovery—The AppLeS agent must dis-cover the resources that are potentially useful to theapplication. This can be accomplished using a list ofthe user’s logins or by using ambient Grid resourcediscovery services [3].

2. Resource Selection—The agent identifies and selectsviable resource sets from among the possibleresource combinations. AppLeS agents typicallyuse an application-specific resource selection modelto develop an ordered list of resource sets [4].Resource evaluation typically employs performancepredictions of dynamically changing system vari-ables (e.g., network bandwidth, CPU load) and/orvalues gathered from previous application execu-tions.

3. Schedule Generation—Given an ordered list offeasible resource sets, the AppLeS agent applies aperformance model to determine a set of candidateschedules for the application on potential targetresources. In particular, for each set of feasibleresources, the agent uses a scheduling algorithm todetermine “the best” schedule for the application onjust the target set (i.e., for any given set of resources,many schedules may be possible).

4. Schedule Selection—Given a set of candidateschedules (and the target resource sets for whichthey have been developed), the agent chooses the“best” overall schedule that matches the user’sperformance criteria (execution time, turnaroundtime, convergence, etc.)

5. Application Execution—The best schedule is de-ployed by the AppLeS agent on the target resourcesusing whatever infrastructure is available. For someAppLeS, ambient services can be used (e.g., Globus[5], Legion [6], NetSolve [7], PVM [8], MPI [9]). Forother AppLeS applications, deployment may beperformed “on the bare resources” by explicitlylogging in, staging data, and starting processes onthe target resources (e.g., via Ssh).

6. Schedule Adaptation—The AppLeS agent can ac-count for changes in resource availability by loopingback to Step 1. Indeed, many Grid resources exhibitdynamic performance characteristics and resourcesmay even join or leave the Grid during theapplication’s lifetime. AppLeS targeting long-runningapplications can then iteratively compute andimplement refined schedules.

Using this approach, we have developed over a dozenAppLeS-enabled applications [2], [10], [11], [12], [13], [14],[15], [16], [17], [4], [18], [19], some of which are used asillustrating examples in the next section.

3 APPLES FUNCTIONALITIES AND APPLESAPPLICATIONS

In almost all cases, the development of an AppLeSapplication has been a joint collaboration between dis-ciplinary researchers and members of the AppLeS projectteam. During such collaborations, the application scientistsprovide an original parallel or distributed application codewhich correctly solves their disciplinary problem. AppLeSresearchers then work with these scientists to modify theapplication so that it can be dynamically scheduled by anAppLeS scheduling agent. The result is a new applicationconsisting of domain-specific components and a customscheduling superstructure that is controlled by the AppleSagent dynamically to effect a schedule for the target Gridenvironment. After developing the AppLeS-enabled appli-cation, the AppLeS team generally performs productionexperiments or simulations to determine whether theadaptive scheduling techniques are performance-efficientcompared to the original code and/or other schedulingalternatives.

While each AppLeS agent is customized for its particularapplication, they share the overall methodology depicted inFig. 1. The following sections describe various AppLeS-enabled applications which illustrate the most importantconcepts of the AppLeS methodology.

3.1 Resource Selection and Simple SARAThe Simple SARA AppLeS [11] demonstrates the AppLeSresource selection step. SARA (Synthetic Aperture RadarAtlas) is an application developed at the Jet PropulsionLaboratory and the San Diego Supercomputer Center whichprovides access to satellite images distributed in variousrepositories [20]. The images are accessed via a Webinterface which allows the user to choose how the imageis processed and the site from which it can be accessed.SARA is representative of an increasingly common class ofimage acquisition applications with such exemplars asDigital Sky [21] or Microsoft’s TerraServer [22].

In the SARA environment, images are typically repli-cated at multiple sites. We developed an AppLeS called“Simple SARA” that focuses only on resource selec-tion—choosing the most performance-efficient site forreplicated image files. Previous to our work on thisapplication, the users were selecting servers “by hand”via the SARA web interface.

On the surface, the selection of storage resources fromwhich to transfer a replicated SARA image file seems easy.Users intuitively pick the closest storage resource geogra-phically or perhaps the storage resource with the greatestmaximum bandwidth. However, network contention mayresult in greatly degraded performance to the mostgeographically proximate site and a site which is farthergeographically or has smaller bandwidth capacity, but isrelatively lightly loaded may actually exhibit greaterperformance.

This was the case in a set of experiments we did withthe Simple SARA AppLeS at Supercomputing ’99 (an

370 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 14, NO. 4, APRIL 2003

Fig. 1. Steps in the AppLeS methodology.

international conference on high-performance computing)in Portland, Oregon. As part of a demonstration system,we placed replicated image files at the Oregon GraduateInstitute (OGI)—less than 10 miles from the conferencefloor, the University of California, San Diego (UCSD)—-over 1,000 miles from the conference floor, and theUniversity of Tennessee, Knoxville (UTK)—over 2,000miles from the conference floor. The Simple SARAAppLeS used the Network Weather Service (NWS) [23],[24], a monitoring and forecasting facility commonly usedby AppLeS agents to provide dynamic predictions of end-to-end available bandwidth. Available bandwidth predic-tions were used to estimate data transfer times.

The results shown in Fig. 2 indicate that, during ourparticular set of experiments (run during the conferenceduring the day), minimal data transfer time was neverachieved by transferring files to the conference exhibitionfloor in Portland from the most geographically proximatesite (OGI in Portland). The best performance was generallyachieved by transferring the image from UCSD andoccasionally achieved by transferring the image from UTK.

The Simple SARA AppLeS uses NWS predictions ofAvailable Bandwidth in order to rank the available dataservers. Therefore, the AppLeS agent experiences thenetwork performance as the user does and chooses the bestdata server consistently.

3.2 Performance Modeling and Jacobi2D

The Jacobi 2D AppLeS [2] provides a good demonstration ofthe development of an application-specific performancemodel which is used to generate a good candidate schedulefor a given set of feasible resources. We have since thenenhanced this performance model for Jacobi and for otherapplications [14], [4], but, for simplicity, we describe hereour original model.

Jacobi 2D is a regular iterative code which continuouslyupdates a 2D matrix within a loop body between globalerror-checking stages. Computation consists of an update toeach matrix entry based on the values of each of theneighbors of the entry in a 4-point stencil. In particular, theJacobi 2D main loop is as follows:

1. Loop until convergence2. For all matrix entries Ai;j

3. Ai;j 14 ðAi;j þAiþ1;j þAiÿ1;j þAi;jþ1 þAi;jÿ1Þ

4. Compute local error

Jacobi 2D is a data-parallel program, so the key toperformance is the distribution of data to processors. TheJacobi 2D AppLeS divides the data matrix into strips whosewidths correspond to the predicted capacity of each targetcomputing resources. The performance model for Jacobi is:

81 � i � p Ti ¼ Areai �Operi �AvailCPUi þ Ci;where p is the number of processors in the target resourceset, Areai is the size of the strip allocated to processor Pi,Operi is the dedicated execution time to compute onematrix entry, AvailCPUi is a prediction of the percentage ofavailable CPU for processor i as provided by the NWS, andCi is a prediction of the communication time betweenprocessor Pi and its neighbor(s) as provided by the NWS.

A schedule can then be computed from this perfor-mance model by solving the time-balancing equations:T1 ¼ T2 ¼ . . . ¼ Tp, for all unknown Areai. These equa-tions ensure that all processors are synchronized at theend of each iteration, thereby minimizing idle time.Simple memory constraints and the requirement that allAreai must be positive are used to filter infeasibleschedules.

Fig. 3 shows application execution times obtained on anondedicated network of workstations spanning machinesin the Computer Science and Engineering Department(CSE) and the San Diego Supercomputer Center (SDSC) atUCSD. We ran the Jacobi 2D application for increasingproblem sizes. First, we observe that our simplistic modelaccurately predicts execution performance in a contendedenvironment. The graph also plots results for applicationexecutions with a uniform, compile-time partitioning ofthe application (what a user might do in practice). Themain observation is that adaptive runtime schedulingoutperforms compile-time scheduling. An interestingphenomenon is the spike that occurred during theexperiments for problem size 1; 900� 1; 900. The spikeoccurred when a gateway on the platform went down.Using the NWS, the Jacobi 2D AppLeS perceives thegateway to be unacceptably slow and assigns strips toCSE or SDSC, but not both, whereas it was usingmachines at both sides before the failure. The uniformcompile-time partitioning, because it did not use dynamicparameterizations of available load and bandwidth,suffered a large performance hit.

3.3 Scheduling Generation and ComplibKey to the AppLeS approach is the ability to generate aschedule that not only considers predicted expectedresource performance, but also the variation in thatperformance. A resource with a large expected performancethat also exhibits a wide performance variation might be“worth” less to an application than one with a lower butmore predictable performance profile.

We exploited this circumstance to build an AppLeS forComplib [25]. Complib is a computational biology applica-tion that compares a library of “unknown” sequencesagainst a database of “known” sequences using the FASTAscoring method. Complib is representative of a large andincreasingly popular class of applications.

An obvious approach is to implement a distributedversion of Complib with the well-known master/workerprogramming model: The master dispatches work in fixed-size work-units to workers in a greedy fashion. This istypically called self-scheduling [26] as it naturally balancesthe workload. Overhead is incurred each time the mastersends work to a worker and each time a worker sends

BERMAN ET AL.: ADAPTIVE COMPUTING ON THE GRID USING APPLES 371

Fig. 2. Simple SARA experiments at Supercomputing ’99. Data transfers

from each site every three minutes during three hours at the conference.

results to the master. This overhead can be reduced byenlarging the size of the work-unit, but this comes at thecost of possible load imbalance since faster workers maywait for slower ones at the end of application execution. Anumber of approaches have been proposed to dynamicallydecrease work-unit sizes throughout application executionin order to mitigate overhead and load imbalance [27], [28].

Alternatively, the source and target sequence librariescan be partitioned among the available processors beforeexecution begins. This is the static partitioning approach thatwas used in the Jacobi 2D example in the previous section.This approach leads to minimal overhead. However, ifexact execution times for each partition cannot be predicted,slower processors may be assigned too much work or fasterones too little, causing potentially high load imbalance.

To combine the benefits provided by both self-schedul-ing and static partitioning, the AppLeS agent for Complibuses predictions of processor speed and network perfor-mance, as well as estimates of the uncertainty in thesepredictions, to compute the dependable performance avail-able to the application. To do so, it consults the NWS [23],[24] to obtain up-to-date predictions of future performanceand the prediction error associated with each prediction.Using a multiplicative factor of the error, it then computes aminimum predicted performance level that each resource is

likely to exceed ([10] describes the calculation of depend-able performance more completely).

When the AppleS-enabled Complib application is exe-cuted, the agent first uses the dependable performance topartition one portion of the two-dimensional score arraystatically, before execution begins, in proportion to therelative dependable computational powers of the proces-sors that are available. The remaining work is self-scheduled during execution. Fig. 4 depicts this schedulingtechnique. In effect, the agent automatically tunes eachexecution based on point-valued predictions of perfor-mance and dynamically generated prediction errors.

Fig. 5 compares the average execution times over 30back-to-back executions of Complib on three differentproblem sizes. The small, medium, and large problem sizescompared 20 unknown sequences to libraries of 10,000,32,000, and 120,000 known sequences, respectively. Allexperiments used a heterogeneous pool of nondedicatedmachines: two four-processor Sun Enterprise servers, sixstand-alone Sun workstations of various speeds, and a 12-processor Sun SMP located at SDSC, UCSD, and the SAICcorporation in Arlington, Virginia.

From Fig. 5, it is clear that the AppLeS agent achievessignificantly better execution times as the problem size


Fig. 4. Dependable performance is used to partition as much of the

workload as possible while the remainder is self-scheduled.

Fig. 3. Correlation of Jacobi 2D performance model and Jacobi 2D execution performance.

Fig. 5. AppLeS partitioning/self-scheduling, AppLeS partitioning only,

customized, hand-tuned self-scheduling.

scales. More surprisingly, the original designers of Complibhad developed a specially tuned implementation of self-scheduling (the third bar in each comparison) for theapplication. The AppLeS agent was able to outperform this“hand-tuned” scheduler by more than a factor of 2.5.

3.4 Schedule Adaptation and MCellOur scheduling work with the MCell application is a goodexample of how scheduling decisions can be adaptivelyrefined throughout application execution with an AppLeSagent. MCell is a computational neuroscience applicationthat studies biochemical interactions within living cells atthe molecular level [29], [30]. MCell experiments areindependent of one another; however, large input files areoften shared by whole subsets of experiments and must bestaged in order for the simulation to execute efficiently. Thisis depicted in Fig. 6. In addition, large numbers ofmoderately sized output files (e.g., totaling hundreds ofGigabytes) must also be iteratively postprocessed.

MCell executions can span several hours/days as MCellresearchers scale simulations to tens or hundreds ofthousands of tasks. Therefore, it is necessary to performschedule adaptation throughout application execution inorder to tolerate changes in resource availabilities.

The problem of scheduling sets of independent tasksonto heterogeneous sets of processors is NP-hard [31] andsubstantial research work has been devoted to designsuitable heuristics. The self-scheduling approach [28], [27],[31], which we discussed in Section 3.3, is inherentlyadaptive. Its main disadvantage is that no planning isperformed because of the greedy scheduling approach. Inparticular, in the case of MCell, self-scheduling does notgenerally lead to good reuse of shared input files. Analternative is to use list-scheduling heuristics [32], [33]. In ourwork with MCell, we extended these heuristics to take intoaccount data transfer costs and data reuse. Then, usingsimulation, we compared several common heuristics: Min-min, Max-min, and Sufferage [33]. We also developed a newheuristic, XSufferage, which extends the sufferage heuristicsto take advantage of computational environments withstorage devices that can be shared by subsets of thecomputational resources. We found that XSufferage out-performs competing heuristics on average and that, ingeneral, list-scheduling outperforms self-scheduling for anapplication such as MCell (see [34]).

We also found that the list-scheduling approach issensitive to performance prediction errors. This is clearlya problem for long-running applications as performancepredictions performed at the onset of the application do nothold throughout execution. Note that the self-schedulingapproach does not make use of performance predictions atall. We implemented a version of our list-schedulingalgorithm in which the schedule is recomputed periodically

(at so-called scheduling events). In simulation, we saw thatadaptation makes it possible for list-scheduling algorithmsto tolerate performance prediction errors and, overall,outperform the self-scheduling approach. This is an im-portant result: Via schedule adaptation, it is possible to usesophisticated scheduling heuristics for Grid environmentsin which resource availabilities change over time. Wefurther validated this result for MCell on a real testbed.

In [15], we compared the XSufferage and a self-scheduling workqueue algorithms for an MCell simulationconsisting of 1,200 experiments and sharing input filesranging from one to 100 MBs. The testbed consisted ofclusters of workstations at UCSD (eight hosts), UTK(16 hosts), and TITECH (32 hosts, Japan). The executiontimes of the MCell simulation were compared under fourdifferent data location scenarios. In all scenarios, all inputdata is available in local storage at the Japanese site.Average execution times over 35 repeated experiments areshown in Fig. 7, including error bars. The experimentsdemonstrate that when file transfer time is not an issue, asin scenario (a) where all files staged everywhere, thatworkqueue and the predictive heuristic both perform well.Scenarios (b), (c), and (d) correspond to situations in whichfewer files are replicated. In scenario (d), input files are onlyavailable in Japan. One can see that, as file staging becomesmore crucial, XSufferage performs better than self-schedul-ing as it take into consideration the location of relevant data.

4 FROM CUSTOMIZED APPLES AGENTS TO

REUSABLE SOFTWARE ENVIRONMENTS

Over the years of the AppLeS project, we have workedwith roughly a dozen disciplinary applications [2], [10],[11], [12], [13], [14], [15], [16], [17], [4], [18]. During thecourse of the AppLeS project, we have often beenapproached by application developers asking for AppLeScode so that they could enhance their own application.However, AppLeS agents are integrated pieces of soft-ware in which the application code and the agent arecombined and not easily separated; in particular, it isdifficult to adapt an AppLeS application to create adifferent AppLeS application.


Fig. 6. Overall structure of a distributed MCell application: Input to

simulations are potentially large files that encode cellular configuration

and environmental information. These input files can be shared by

multiple simulations.

Fig. 7. Experiments comparing application execution time with XSuffer-age and workqueue scheduling in four scenarios: (a) all files prestagedeverywhere, (b) 100 MByte and two 20 MByte files also prestaged inTennessee, (c) 100 MByte files prestaged in California, and (d) noprestaged files.

To ease this programming burden, we developed AppLeSTemplates that embody common characteristics from varioussimilar (but not identical) AppLeS-enabled applications.Whereas an AppLeS application integrates an adaptivescheduling agent with the application to form a new, self-scheduling adaptive application, an AppLeS template is asoftware framework developed so that an applicationcomponent can be easily “inserted” in modular form intothe template to form a new self-scheduling application.Each AppLeS template is developed to host a structurallysimilar class of applications. To date, we have developedtwo AppLeS templates—APST (AppLeS Parameter SweepTemplate) [35], [15], which targets parameter sweepapplications, and AMWAT (AppLeS Master-Worker Appli-cation Template) [36], which targets master/worker appli-cations. In addition, we have developed a supercomputerAppLeS (SA) [13], [37] for scheduling moldable jobs onspace-shared parallel supercomputers. We describe thesethree projects in what follows.

4.1 APSTThe MCell application discussed in Section 3.4 is represen-tative of an entire class of applications: Parameter SweepApplications (PSAs). These applications are structured assets of computational tasks that are mostly independent:There are few task synchronization requirements or datadependencies among tasks. In spite of its simplicity, thisapplication model arises in many fields of science andengineering, including bioinformatics [38], [39], [40], parti-cle physics [41], [42], discrete-event simulation [43], [44],computer graphics [45], and in many areas of biology [46],[47], [29]. APST is a Grid application execution environ-ments targeted to PSAs.

PSAs are commonly executed on a network of work-stations. Indeed, it is straightforward for users to launchseveral independent jobs on those platforms, for instance,via ad hoc scripts. However, many users would like to scaleup their PSAs and benefit from the vast numbers ofresources available in Grid platforms. Fortunately, PSAsare not tightly coupled as tasks do not have stringentsynchronization requirements. Therefore, they can toleratehigh network latencies such as the ones expected on wide-area networks. In addition, they are amenable to straight-forward fault tolerance mechanisms as tasks can berestarted from scratch after a failure. The ability to applywidely distributed resources to PSAs has been recognizedin the Internet computing community (e.g., SETI@home[48]). There are two main challenges for enabling PSAs atsuch a wide scale: making application execution easy for theusers and achieving high performance. APST addressesthese two challenges by providing transparent deploymentand automatic scheduling of both data and computation.

When designing and implementing APST, the focus wason the following basic principles and goals:

Ubiquitous Deployment: APST users should be able todeploy their applications on as many resources aspossible. Therefore, APST supports a variety of middle-ware services for discovering, using, and monitoringstorage, compute, and network resources.

Opportunistic Execution: Another principle behind APSTis that no specific service is required. For instance, ifservices for resource monitoring are deployed andavailable to the user, then they can be used by ascheduler within APST for making more informedscheduling decisions. However, if no such service isavailable, APST will still function, but will probablyachieve lower performance.

Simple User Interface: APST uses a simple, XML-basedinterface that can be used from the command-line orfrom scripts. This interface can be easily integrated withmore sophisticated interfaces in other Grid projects [49],[50], [51].

Resilience: Grid resources are shared and federated and aretherefore prone to failures and downtimes. APSTimplements simple fault-detection restart mechanismsfor application tasks. To prevent crashes of APST itself,the software uses a checkpointing mechanism to easilyrecover with minimal loss for the application.

4.1.1 The APST SoftwareThe APST software runs as two distinct processes: adaemon and a client. The daemon is in charge of deployingand monitoring applications. The client is essentially aconsole that can be used periodically, either interactively orfrom scripts. The user can invoke the client to interact withthe daemon to submit requests for computation and checkon application progress.

Fig. 8 shows the architecture of the APST software. Thecomputing platform consists of storage, compute, andnetwork resources depicted at the bottom of the figure.Those resources are accessible via deployed middlewareservices (e.g., Grid services as shown on the figure). Thecentral component of the daemon is a scheduler, whichmakes all decisions regarding the allocation of resources toapplication tasks and data. To implement its decisions, thescheduler uses a data manager and a compute manager. Bothcomponents use middleware services to launch andmonitor data transfers and computations. In order to makedecisions about resource allocation, the scheduler needsinformation about resource performance. As shown in thefigure, the scheduler gathers information from threesources. The data manager and the compute manager bothkeep records of past resource performance and provide thescheduler with that historical information. The third source,the metadata manager, uses information services to activelyobtain published information about available resources(e.g., CPU speed information from MDS [52]). A predictor,not shown on the figure, compiles information from those


Fig. 8. Architecture of the APST software.

three sources and computes forecasts, using techniquesfrom NWS [23]. Those forecasts are then used by APST’sscheduling algorithms. The cycle of control and databetween the scheduler and the three managers is key foradaptive scheduling of PSAs onto Grid platforms. Theadaptive scheduling strategies used by APST are inheritedfrom our work with MCell as described in Section 3.4.

The current APST implementation can make use of anumber of middleware services and standard mechanismsto deploy applications. We provide a brief description ofthose capabilities.

Launching Application tasks: APST can launch applicationtasks on the local host using fork. Remote hosts can beaccessed via ssh, Globus GRAM [53], and NetSolve [7].The ssh mechanism allows for ssh-tunneling in order togo through firewalls and to private networks. APSTinherits the security and authentication mechanismsavailable from those services (e.g., GSI [54]), if any.APST can launch applications directly on interactiveresources and can start jobs via schedulers such as PBS[55], LoadLeveler [56], and Condor [57].

Moving and Storing Application Data: APST can read,copy, transfer, and store application data among storageresources with the following mechanisms. It can use cp

to copy data between the user’s local host to storageresources that are on the same Network File System; datacan also be used in place. APST can also use scp, FTP,GASS [58], GridFTP [59], and SRB [60]. APST inheritsany security mechanisms provided by those services.

Discovering and Monitoring Resources: APST can obtainstatic and dynamic information from services such asMDS [52] and NWS [23]. APST also learns aboutavailable resources by keeping track of their pastperformance when computing application tasks ortransferring application data.

A number of research articles describing the APST workhave been published: Casanova et al. [34] presents anevaluation of APST’s scheduling heuristics, Casanova et al.[15] contains experimental results obtained on a Gridplatform spanning clusters in Japan, California, andTennessee, Casanova et al. [61] describes the use of APSTspecifically for a computational neuroscience application,and Casanova and Berman [62] discusses the latest APSTimplementation as well as usability issues. APST is relatedto other Grid projects such as ILAB [50], Nimrod/G [51],and Condor [57], which also provide mechanisms forrunning PSAs and, more generally, to projects in industry[63], [64], [65] that aim at deploying a large number of jobson widely distributed resources.

4.1.2 APST UsageAPST started as a research prototype for exploring adaptivescheduling of PSAs on the Grid platform. Since then, it hasevolved into a usable software tool that is gainingpopularity in several user communities. The first applica-tion to use APST in production was MCell. Since then,APST has been used for computer graphics applications[45], discrete event simulations [43], [44], and bioinformaticsapplications [40], [38], [39]. There is growing interest in thebioinformatics community as biological sequence matchingapplications all fit under the PSA model.

A striking realization is that, at this stage of Gridcomputing, users are more concerned with usability thanwith performance. Many disciplinary scientists are stillrunning their applications on single workstations. It wassurprising to realize that, even for parallel applications assimple as PSAs, there are still many hurdles for users toovercome. APST provides a good solution because it does

not require modification of the application, because itrequires only a minimal understanding of XML, andbecause it can be used immediately with ubiquitousmechanisms (e.g., ssh and scp). In addition, users can easilyand progressively transition to larger scale platforms onwhich more sophisticated Grid services are required.

4.2 AMWATWhile APST targets large-scale parameter sweep applica-tions with potential data locality issues, the AppLeSMaster-Worker Application Template, or AMWAT, targetsdeployment of small and medium-scale Master-Worker(MW) applications. Another difference between APST andAMWAT is that AMWAT provides an API for users touse when writing their applications, whereas APSTdeploys existing applications without any modificationto their code. MW applications traditionally have a singlemaster process which controls the flow of computationthat is performed on one or more remote workerprocesses. An MW organization is one of the mostcommonly used approaches for parallelizing computa-tions and is particularly well-suited for problems whichcan be easily subdivided into independent tasks forcomputation on distributed resources such as thoseprovided by Grid environments.

The focus of AMWAT is to simultaneously address threeproblems in developing and deploying MW applications forGrid environments: 1) reducing the basic costs of develop-ing Grid codes, 2) ensuring ready portability to manydifferent platforms, and 3) providing scheduling capabil-ities which can deliver consistently good performanceunder a wide variety of conditions for MW applications.AMWAT accomplishes these goals by providing much ofthe functionality required to build MW applications in theform of portable and reusable modules; thereby allowingapplication developers to concentrate their efforts onproblem-specific details. Fig. 9 shows the basic organizationof the AMWAT application framework.

Application-specific functions are provided by devel-opers through a set of 15 application activity functionsspecified in the Application Template module shown in Fig. 9.Application portability is provided by implementingcommon interfaces to various Grid services such asinterprocess communication and process invocation in thePortable Services module shown in Fig. 9. As an example, acommon communication interface provides flexible accessto established communication methods such as Unix-stylesockets, System V IPC shared memory, PVM [8], and MPI[9]. Support for delivering consistent application perfor-mance is provided by both general and specialized MWscheduling functions contained in the Scheduling module ofFig. 9. Additional details for each of the AMWAT modulescan be found in [36]. AMWAT is strongly related to theCondor Master-Worker project [66], [67] which also


Fig. 9. Components of the AMWAT application framework.

investigates scheduling issues for master/worker computa-tion on Grid platforms. A thorough discussion of relatedwork can be found in [36].

Unique to the AMWAT scheduling approach is theattention given to addressing performance bottleneckscaused by the runtime interaction of application demandsand deliverable resource capabilities. As part of the work increating AMWAT, we developed a work-flow model ofMW application performance and applied this model toderive an approach for selecting performance-efficient hostsfor both the master and worker processes in a MWapplication [68]. This approach accounts for the effects ofboth computation and communication performance con-straints on MW performance in dynamic heterogeneousenvironments.

We have also investigated the role of work distributionstrategies in allowing MW applications to cope withdegrees of variability in both application characteristicsand resource behavior [36]. Experimental results showedthat no single strategy performed best under differentenvironmental conditions, even when running the sameapplication. As an example, Fig. 10 shows observed effectson execution time due to both the choice of workdistribution strategy used and the granularity of workbeing distributed for the Povray ray-tracing program,running in a wide-area network environment connectingworkstations at UCSD and UTK. In addition to a simpleone-time fixed allocation (FIXED) strategy, other testedwork distribution strategies include: Self Scheduling (SS)[69], Fixed Size Chunking (FSC) [70], Guided Self Schedul-ing (GSS) [71], Trapezoidal Self Scheduling (TSS) [72], andFactoring (FAC2) [73]. FIXED, SS, and FSC are examples ofallocation strategies which apply the same allocation blocksizes throughout an application run, while GSS, TSS, andFAC2 are examples of strategies which utilize decreasingblock sizes as an application progresses. Each strategydiffers from the other strategies in the manner by whichinitial and subsequent block sizes are determined. TheAMWAT Scheduling module has been designed to providea selectable choice of work distribution strategies to allow

MW applications developed with AMWAT increasedflexibility in adapting performance in response to specificconditions encountered.

AMWAT has been ported to a variety of computingplatforms, including workstations running Linux, SunSolaris, IBM AIX, SGI IRIX, and HP HPUX operatingsystems, as well as high-performance supercomputers suchas the Cray T3E and IBM Blue Horizon located at SDSC, andtested with a number of applications [74], [75], [76], [12].

4.3 SAWhile APST and AMWAT target Grid environments,Supercomputer AppleS (SA) targets space-shared supercom-puter environments [77], [78], [37], showing that applicationlevel scheduling can be useful in general for user-directedscheduling. SA is a generic AppLeS which promotes theperformance of moldable jobs (i.e., jobs that can be executedwith any of a collection of possible partition sizes) in abatch-scheduled, space-shared, back-filling environment.Such environments are common in production super-computer centers and include MPPs scheduled by EASY[79], the Maui Scheduler [80], and LSF [81].

SA chooses which partition size to use for submitting amoldable job request. This decision is important because itaffects the job’s turn-around time (the time elapsed betweenthe job’s submission and its completion). Sometimes it isbetter to use a small request which is going to executelonger, but does not wait too much in the job queue. Othertimes, a large request delivers the best turn-around time(when the queue is short, for example).

The user provides SA with a set of possible partitionsizes that can be used to submit a given moldable job. SAuses simulations to estimate the turn-around time of eachpotential request based on the current state of the super-computer and then forwards to the supercomputer therequest with the smallest expected turn-around time.

SA does not always select the best request because theexecution times of the jobs already in the system are notknown (request times are used as estimates) and future


Fig. 10. Effects of work distribution strategies on Povray performance in a WAN environment.

arrivals can affect jobs already in the system. However, SAchooses close to an optimal request for most jobs and itspick is generally considerably better than the user’s choice.In order to quantify the improvement due to the use of SA,we compare the turn-around time obtained by SA’s choiceagainst the turn-around time attained by the user’s choiceand also against the best turn-around time among allchoices SA had available to choose from. We used foursupercomputer workload logs in our experiments [77]. Ineach experiment, a job is randomly chosen and subjected toa model that generates alternative requests for the job [82].The workload is then simulated 1) using the users request tosubmitted the job, 2) letting SA select which request to use,and 3) using all possible requests, which enables us todetermine the best request. This experiment was repeated360,000 times. Since job execution time and turn-aroundtime are so widely distributed in supercomputers [83], wechose not to use summarizing statistics (like the mean) tocompare results. Instead, we use the whole distribution of therelative turn-around times to gauge the performance of SA.Relative turn-around times depict the turn-around time ofSA and the best request as a fraction of the turn-aroundtime of the users choice. More precisely, the relative turnaround time of SA, relttSA, is given by relttSA ¼ ttSA=ttuser.Similarly, the best relative turn-around time is given byrelttbest ¼ ttbest=ttuser. Fig. 11 shows cumulative distribu-tions of the relative turn-around times of SA and the ”best”request [77]. Note that, due to the definition of relative turn-around time, values smaller than 1 indicate that SA (or bestrequest) had smaller turn-around times than the usersrequest. Values greater than 1 indicate that the users choiceperformed better. Values equal to 1 show that SA (or bestrequest) achieved exactly the same turn-around time.Therefore, as can be seen in Fig. 11, SA improves theperformance of 45.8 percent of the jobs, sometimesdramatically. 45.3 percent of the jobs do not experienceany performance increase. This is due to the fact that, inthese simulations, SA was not offered with many choices forpartition size, conservatively modeling observed behaviors[77], [82]. For 8.8 percent of the jobs, the user choice for therequest resulted in better turn-around time than the requestselected by SA. More details about SA and these experi-ments can be found in [77], [78], [37].

5 RELATED WORK

A number of research efforts target the development ofapplication schedulers for dynamic, heterogeneous com-puting environments. It is often difficult to make head-to-head comparisons between distinct efforts as schedulers aredeveloped for particular system environments, languagerepresentations, and application domains. In this section,we review a number of successful projects [84], [85], [86],[87], [88], [89] and highlight differences with the AppLeSapproach in terms of computing environment, program model,performance model, and scheduling strategy.

Several projects provide a custom runtime system withfeatures that can be used directly by a scheduler. Forinstance, MARS [84], [90] and Dome [89] both provideruntime systems that support task checkpointing andmigration. The schedulers use migration to achieve load-balancing. By contrast, AppLeS does not provide a runtimeenvironment. Instead, it uses what computing infrastruc-ture is available to the user (e.g., Globus [5], Legion [6],NetSolve [7], MPI [9], etc.) along with the NWS [23] forresource monitoring and performance prediction. Thesesystems usually do not offer all the capabilities that couldbe provided by a custom runtime system designed withload-balancing in mind (e.g., task migration in Dome andMARS). However, AppLeS capitalizes on emerging stan-dards (e.g., Globus) and is directly usable by users who arealready familiar with existing environments.

Many projects impose structural requirements for theapplication. For instance, MARS [84] targets SPMD programthat consist of phases within which the execution profileremains the same over several runs. For each phase, one canthen find an optimal task-to-processor mapping. VDCE [85]and SEA [86] target applications structured as dependencygraphs with coarse-grained tasks (calls to functions frommathematical libraries in VDCE, data-flow-style program-ming in SEA). IOS [87] targets real-time, fine-grained,iterative, image recognition applications that are alsostructured as dependency graphs. Dome [89] and SPP(X)[88] provide a language abstraction for the application


Fig. 11. Distribution of relative turn-around time for SA and the best request.

program, which is compiled into a low-level taskdependency graph representation automatically. Domeimposes an SPMD structure, whereas SPP(X) uses thetraditional task dependency graph model. AppLeS fo-cuses on coarse-grain applications. The program model isthat of communicating tasks that are described in termsof their resource requirements. Originally, AppLeS didnot impose any restriction on the programming modeland every instance of AppLeS used its own model. Thisallowed the AppLeS methodology to be applicable tomany application domains in various computing environ-ment settings but limited possible reuse for otherapplications. The templates presented in Section 4 providesoftware for several classes of applications to readilybenefit from the AppLeS methodology.

A performance model provides an abstraction of thebehavior of an application on an underlying set of hardwareresources. The common approach is to parameterize theperformance model by both static and dynamic informationconcerning the available resources. At one end of thespectrum are performance models that are derived from theprogram model. SPP(X) [88] and VDCE [85] derive theirperformance models directly from task dependency graphs.MARS [84] uses a cost model between each program phaseto assess whether it is worth checkpointing and migrating asubset of the application tasks. The cost model is based onstatistics of previous executions of the application fromwhich MARS extracts characteristic load distribution andcommunication patterns. Dome [89] achieves load balan-cing thanks to a sequence of short-term adjustments of thedata distribution among the processors. These adjustmentsare based on the observed “computational rate” of eachprocessor. IOS [87] associates a set of algorithms to eachfine-grain task in the program graph and evaluatesprestored offline mappings of the graph onto the resources.SEA [86] uses its data-flow-style program graph todetermine which tasks are ready for execution. By contrast,AppLeS assumes that the performance model is provided bythe user. Current AppLeS applications rely on structuralperformance models [91], which compose performanceactivities into a prediction of application performance.

Almost all the aforementioned projects do not providemuch latitude for user-provided scheduling policies orperformance criteria: Minimization of execution time is theonly goal. Scheduling in MARS [84] and Dome [89] is donewith iterative task/data redistributions decisions. VDCE[85] uses a list scheduling algorithm, whereas SEA [86] usesan expert system that uses the data-flow representation ofthe program to schedule tasks on the fly. IOS [87] uses anovel approach that uses offline genetic algorithms toprecompute schedules for different program parameters.Some of these schedules are then selected during applica-tion execution. The default AppLeS scheduling policy is toperform resource selection as an initial step and to choosethe best schedule among a set of candidates based on theuser’s performance criteria. Since AppLeS uses a verygeneral program and performance model, there is no singleAppLeS scheduling algorithm, but rather a series of instantia-tions of the AppLeS paradigm. Some of these instantiationshave been reviewed in Section 3. AppLeS is thereforeapplicable to a wide range of applications with variousrequirements and performance metrics and AppLeS sche-dulers can use arbitrary scheduling algorithms that are best

for their target domain. This was successfully demonstratedin the AppLeS template effort.

Finally, the GrADS project [92] has focused on develop-ing a comprehensive and adaptive Grid programmingenvironment. The GrADS software, GrADSoft, is based onmany of the AppLeS principles and generalizes the AppLeSmethodology.

6 ONGOING WORK AND NEW DIRECTIONS

The AppLeS project has demonstrated that taking intoaccount both application and platform-specific informationis key to achieving performance for distributed applicationsin modern computing environments such as the Grid [1]. Ourcurrent and future works are on two fronts: 1) disseminateand deploy AppLeS methodology and technology and2) extend AppLeS methodology to new types of applicationsand computing platforms.

In terms of dissemination and deployment, our first stepwas to develop the AppLeS templates described in Section 4.We are currently disseminating those templates in severaluser communities for different types of applications andcomputing environments. The feedback that we obtain fromthese users is invaluable. It allows us to better understandthe current needs of different scientific communities and tosteer AppLeS template software development efforts ade-quately. Software distributions and documentations areavailable for APST and AMWAT at [93]. AppLeS metho-dology is also being integrated into software from theGrADS [92] project. This project is a multi-institution effortto provide a software development environment for nextgeneration Grid applications. In collaboration with re-searchers at Rice, we are developing a software architectureto facilitate information flow and resource negotiationamong applications, libraries, compilers, schedulers, andthe runtime system. In this context, the AppLeS approach isused for application scheduling and rescheduling.

We are extending the AppLeS approach for three severalnew classes of application. First, as part of the VirtualInstrument project [94], we are exploring application sche-duling in computational steering scenarios. This project is amulti-institution collaboration and is focused on providing asoftware environment for the MCell application (see Section3.4) that enables computational steering. In this project, notonly does the environment exhibit dynamic behaviors, butthe user can steer the application as it runs in order to changeits overall computational goals. We are currently studyinghow this new kind of dynamic phenomenon impactsapplication-level scheduling and we are determining whichscheduling strategies will be applicable [95].

Second, we are exploring application tunability in con-junction with application-level scheduling. Tunability al-lows users to express tradeoffs between several aspects ofan application execution. Typical tradeoffs are between“quality” of application output and resource usage and weare studying tunability in the context of online paralleltomography [18]. We are extending the AppLeS methodol-ogy to assist the user in choosing an appropriate trade-off.

Third, we are investigating application-level schedulingfor nondeterministic applications. In particular, we areconsidering applications that simulate the behavior of largepopulations that consist of well-understood entities (e.g.,ecology or biology simulations). These applications exhibit


emergent behaviors throughout execution and it is verydifficult to define a scheduling strategy that will be effectivethroughout an entire run. As a first approach, we aredeveloping adaptive scheduling techniques that are basedon evolutionary algorithms and on rescheduling. In pre-vious AppLeS work, we mostly focused on accounting fordynamic behavior of the computing environment. Bycontrast, these three extensions to the AppLeS methodologyall address some dynamic aspect of the application. Currenttrends indicate that next generation Grid applications willbe increasingly dynamic and tunable and we expect ourcurrent work to be critical for addressing upcoming Gridcomputing challenges.

We are also studying a class of computing platforms thathas recently become popular: peer-to-peer and global comput-ing environments. These environments have tremendouspotential as they gather thousands of otherwise idle PCsworld-wide. Even though resources are abundant, they arevolatile. Furthermore, little information is available con-cerning resource behaviors. As a result, only embarrass-ingly parallel applications (e.g., SETI@home [48]) have beentargeted to the peer-to-peer platform. Some of our currentwork is investigating whether it is possible to executeapplications with a more complex structure on that plat-form [96]. We are targeting bioinformatics applications and,more specifically, gene sequence-matching algorithms thatcan be implemented in parallel with data dependencies.This departs from original AppLeS work on the Grid asscheduling must be done with little information concerningthe available resources. Recently published work [97]identifies similarities and differences between Grid com-puting and peer-to-peer computing. The current trend is toprovide some unifying framework for wide-area computingand our work on extending AppLeS methodology to peer-to-peer computing will be eminently applicable in thatcontext.

7 SUMMARY

In this article, we have described the AppLeS (ApplicationLevel Scheduling) project. The project focuses on providingmethodology, application-development software, and ap-plication execution environments for adaptively schedulingapplications in dynamic, heterogeneous, multiuser Gridplatforms. In Section 2, we described the general AppLeSmethodology and highlighted the six main stages of ourapproach to application-level adaptive scheduling. InSection 3, we described in details each AppLeS functional-ities and provided explanatory examples from our workwith actual Grid applications. This corresponds to the firstgeneration AppLeS work in which we defined and validatedour methodology. In order to deploy that methodology aspart of reusable software tools, we designed and imple-mented generic AppLeS templates. Those templates weredescribed in Section 4. The APST and AMWAT templates(Sections 4.1 and 4.2) each target a specific class ofapplications and are currently being used by variousapplication groups. The SA template (Section 4.3) can beused as part of supercomputer schedulers for betterresource usage. We compared the AppLeS methodologyto related work in Section 5. As seen in Section 6, our newdirections extend the AppLeS methodology to new classesof applications and platforms that will be critical for nextgeneration Grid computing.

ACKNOWLEDGMENTS

The authors would like to acknowledge the following people

for their help and support throughout the different stages of

the AppLeS project: T. Bartol, A. Downey, M. Ellisman,

M. Farrellee, G. Kremenek, A. Legrand, S. Matsuoka,

R. Moore, H. Nakada, O. Sievert, M. Swany, A. Takefusa,

N. Williams, and the UCSD/CSE and SDSC support staff.

They also wish to thank the reviewers for their insightful

comments which have greatly contributed to improving this

article.

REFERENCES

[1] The Grid: Blueprint for a New Computing Infrastructure. I. Foster andC. Kesselman, eds., San Francisco, Calif.: Morgan KaufmannPublishers, 1999.

[2] F. Berman, R. Wolski, S. Figueira, J. Schopf, and G. Shao,“Application Level Scheduling on Distributed HeterogeneousNetworks,” Proc. Supercomputing 1996, 1996.

[3] C. Czajkowski, S. Fitzgerald, I. Foster, and C. Kesselman, “GridInformation Services for Distributed Resource Sharing,” Proc. 10thIEEE Symp. High-Performance Distributed Computing, 2001.

[4] H. Dail, H. Casanova, and F. Berman, “A Decoupled SchedulingApproach for the GrADS Environment,” Proc. Supercomputing ’02,Nov. 2002.

[5] I. Foster and K Kesselman, “Globus: A Metacomputing Infra-structure Toolkit,” Int’l J. Supercomputer Applications, vol. 11, no. 2,pp. 115-128, 1997.

[6] A. Grimshaw, A. Ferrari, F.C. Knabe, and M. Humphrey, “Wide-Area Computing: Resource Sharing on a Large Scale,” Computer,vol. 32, no. 5, pp. 29-37, May 1999.

[7] H. Casanova and J. Dongarra, “NetSolve: A Network Server forSolving Computational Science Problems,” Int’l J. SupercomputerApplications and High Performance Computing, vol. 11, no. 3, pp. 212-223, 1997.

[8] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V.Sunderam, PVM: Parallel Virtual Machine. A Users’ Guide andTu-torial for Networked Parallel Computing. Cambridge, Mass.: MITPress, 1994.

[9] M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra,MPI: The Complete Reference. Cambridge, Mass.: MIT Press, 1996.

[10] N. Spring and R. Wolski, “Application Level Scheduling of GeneSequence Comparison on Metacomputers,” Proc. 12th ACM Int’lConf. Supercomputing, July 1998.

[11] A. Su, F. Berman, R. Wolski, and M. Mills Strout, “Using AppLeSto Schedule Simple SARA on the Computational Grid,” Int’l J.High Performance Computing Applications, vol. 13, no. 3, pp. 253-262,1999.

[12] S. Smallen, W. Cirne, J. Frey, F. Berman, R. Wolski, M.H. Su, C.Kesselman, S. Young, and M. Ellisman, “Combining Workstationsand Supercomputers to Support Grid Applications: The ParallelTomography Experience,” Proc. Ninth Heterogeneous ComputingWorkshop, pp. 241-252, May 2000.

[13] W. Cirne and F. Berman, “Adaptive Selection of Partition Size forSupercomputer Requests,” Proc. Sixth Workshop Job SchedulingStrategies for Parallel Processing, May 2000.

[14] H. Dail, G. Obertelli, F. Berman, R. Wolski, and A. Grimshaw,“Application-Aware Scheduling of a MagnetohydrodynamicsApplication in the Legion Metasystem,” Proc. Ninth HeterogeneousComputing Workshop (HCW ’00), May 2000.

[15] H. Casanova, G. Obertelli, F. Berman, and R. Wolski, “TheAppLeS Parameter Sweep Template: User-Level Middleware forthe Grid,” Proc. Supercomputing 2000 (SC ’00), Nov. 2000.

[16] J. Schopf and F. Berman, “Stochastic Scheduling,” Proc. Super-computing 1999, 1999.

[17] J. Schopf and F. Berman, “Using Stochastic Information to PredictApplication Behavior on Contended Resources,” Int’l J. Founda-tions of Computer Science, vol. 12, no. 3, pp. 341-363, 2001.

[18] S. Smallen, H. Casanova, and F. Berman, “Tunable On-LineParallel Tomography,” Proc. Supercomputing ’01, Nov. 2001.

[19] M. Faerman, A. Su, R. Wolski, and F. Berman, “AdaptivePerformance Prediction for Distributed Data-Intensive Applica-tions,” Proc. Supercomputing ’99, Nov. 1999.


[20] Caltech’s synthetic aperture radar atlas project, http://www.cacr.caltech.edu/\~roy/sara/index.html. year?

[21] Caltech’s digital sky project, http://www.cacr.caltech.edu/digisky, 2002.

[22] Microsoft’s terraserver project, http://terraserver.homeadvisor.msn.com, 2002.

[23] R. Wolski, N. Spring, and J. Hayes, “The Network WeatherService: A Distributed Resource Performance Forecasting Servicefor Metacomputing,” Future Generation Computer Systems, vol. 15,nos. 5-6, pp. 757-768, Oct. 1999.

[24] R. Wolski, “Dynamically Forecasting Network Performance Usingthe Network Weather Service,” Cluster Computing, vol. 1, pp. 119-132, Jan. 1998.

[25] A.S. Grimshaw, E.A. West, and W.R. Pearson, “No Pain andGain!—Experiences with Mentat on Biological Applications,”Concurrency: Practice and Experience, vol. 5, no. 4, July 1993.

[26] T. Hagerup, “Allocating Independent Tasks to Parallel Processors:An Experimental Study,” J. Parallel and Distributed Computing,vol. 47, pp. 185-197, 1997.

[27] S. Flynn Hummel, J. Schmidt, R.N. Uma, and J. Wein, “Load-Sharing in Heterogeneous Systems via Weighted Factoring,” Proc.Eighth Ann. ACM Symp. Parallel Algorithms and Architectures,pp. 318-328, June 1996.

[28] T. Hagerup, “Allocating Independent Tasks to Parallel Processors:An Experimental Study,” J. Parallel and Distributed Computing,vol. 47, pp. 185-197, 1997.

[29] J.R. Stiles, T.M. Bartol, E.E. Salpeter, and M.M. Salpeter, “MonteCarlo Simulation of Neuromuscular Transmitter Release UsingMCell, a General Simulator of Cellular Physiological Processes,”Computational Neuroscience, pp. 279-284, 1998.

[30] J.R. Stiles, D. Van Helden, T.M. Bartol, E.E. Salpeter, and M.M.Salpeter, “Miniature End-Plate Current Rise Times < 100 Micro-seconds from Improved Dual Recordings Can Be Modeled withPassive Acetylcholine Diffusion Form a Synaptic Vesicle,” Proc.Nat’l Academy of Sciences U.S.A., vol. 93, pp. 5745-5752, 1996.

[31] O.H. Ibarra and C.E. Kim, “Heuristic Algorithms for SchedulingIndependent Tasks on Nonindentical Processors,” J. ACM, vol. 24,no. 2, pp. 280-289, Apr. 1977.

[32] R.D. Braun, H.J. Siegel, N. Beck, L. Boloni, M. Maheswaran, A.I.Reuther, J.P. Robertson, M.D. Theys, B. Yao, D. Hensgen, and R.F.Freund, “A Comparison Study of Static Mapping Heuristics for aClass of Meta-Tasks on Heterogeneous Computing Systems,”Proc. Eighth Heterogeneous Computing Workshop (HCW ’99), pp. 15-29, Apr. 1999.

[33] M. Maheswaran, S. Ali, H.J. Siegel, D. Hensgen, and R. Freund,“Dynamic Matching and Scheduling of a Class of IndependentTasks onto Heterogeneous Computing Systems,” Proc. EighthHeterogeneous Computing Workshop (HCW ’99), pp. 30–44 Apr. 1999.

[34] H. Casanova, A. Legrand, D. Zagorodnov, and F. Berman,“Heuristics for Scheduling Parameter Sweep Applications in GridEnvironments,” Proc. Ninth Heterogeneous Computing Workshop(HCW ’00), pp. 349-363, May 2000.

[35] Apst homepage, http://grail.sdsc.edu/projects/apst, 2002.[36] G. Shao, “Adaptive Scheduling of Master/Worker Applications

on Distributed Computational Resources,” Ph.D. thesis, Univ. ofCalifornia, San Diego, May 2001.

[37] W. Cirne, “Using Moldability to Improve the Performance ofSupercomputer Jobs,” Ph.D. thesis, Univ. of California, San Diego,Nov. 2000.

[38] S. Altschul, W. Dish, W. Miller, E. Myers, and D. Lipman, “BasicLocal Alignment Search Tool,” J. Molecular Biology, vol. 215,pp. 403-410, 1990.

[39] S. Altschul, T. Madden, A. Schaffer, J. Zhang, Z. Zhang, W. Miller,and D. Lipman, “Gapped BLAST and PSI-BLAST: A NewGeneration of Protein Database Search Programs,” Nucleic AcidsResearch, vol. 25, pp. 3389-3402, 1997.

[40] R. Durbin, S. Eddy, A. Krogh, and G. Mitchison, BiologicalSequence Analysis: Probabilistic Models of Proteins and NucleicAcids. Cambridge Univ. Press, 1998.

[41] J. Basney, M. Livny, and P. Mazzanti, “Harnessing the Capacity ofComputational Grids for High Energy Physics,” Proc. Conf.Computing in High Energy and Nuclear Physics, 2000.

[42] A. Majumdar, “Parallel Performance Study of Monte-CarloPhoton Transport Code on Shared-, Distributed-, and Distribu-ted-Shared-Memory Architectures,” Proc. 14th Parallel and Dis-tributed Processing Symp., IPDPS ’00, pp. 93-99, May 2000.

[43] H. Casanova, “Simgrid: A Toolkit for the Simulation of Applica-tion Scheduling,” Proc. IEEE Int’l Symp. Cluster Computing and theGrid (CCGrid ’01), pp. 430-437, May 2001.

[44] A. Takefusa, S. Matsuoka, H. Nakada, K. Aida, and U.Nagashima, “Overview of a Performance Evaluation System forGlobal Computing Scheduling Algorithms,” Proc. Eighth IEEE Int’lSymp. High Performance Distributed Computing (HPDC), pp. 97-104,Aug. 1999.

[45] NPACI Scalable Visualisation Tools Webpage, http://vistools.npaci.edu, 2002.

[46] H. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. Bhat, H.Weissig, I. Shingyalov, and P. Bourne, “The Protein Data Bank,”Nucleic Acids Research, vol. 28, no. 1, pp. 235-242, 2000.

[47] A. Natrajan, M. Crowley, N. Wilkins-Diehr, M. Humphrey, A.Fox, and A. Grimshaw, “Studying Protein Folding on the Grid:Experiences Using CHARM on NPACI Resources under Legion,”Proc. 10th IEEE Int’l Symp. High Performance Distributed Computing(HPDC-10), 2001.

[48] SETI@Home project, http://setiathome.ssl.berkeley.edu, 2002.[49] M. Thomas, S. Mock, J. Boisseau, M. Dahan, K. Mueller, and D.

Sutton, “The GridPort Toolkit Architecture for Building GridPortals,” Proc. 10th IEEE Int’l Symp. High Performance DistributedComputing (HPDC-10), Aug. 2001.

[50] M. Yarrow, K. McCann, R. Biswas, and R. Van der Wijngaart, “AnAdvanced User Interface Approach for Complex Parameter StudyProcess Specification on the Information Power Grid,” Proc. GRID2000, Dec. 2000.

[51] D. Abramson, J. Giddy, and L. Kotler, “High PerformanceParametric Modeling with Nimrod/G: Killer Application for theGlobal Grid?” Proc. Int’l Parallel and Distributed Processing Symp.(IPDPS), pp. 520-528, May 2000.

[52] K. Czajkowski, S. Fitzgerald, I. Foster, and C. Kesselman, “GridInformation Services for Distributed Resource Sharing,” Proc. 10thIEEE Symp. High-Performance Distributed Computing, 2001.

[53] K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W.Smith, and S. Tuecke, “A Resource Management Architecture forMetacomputing Systems,” Proc. IPPS/SPDP ’98 Workshop JobScheduling Strategies for Parallel Processing, pp. 62-82, 1998.

[54] I. Foster, C. Kesselman, G. Tsudik, and S. Tuecke, “A SecurityArchitecture for Computational Grids,” Proc. Fifth ACM Conf.Computer and Comm. Security, pp. 83-92, 1998.

[55] The Portable Batch System Webpage, http://www.openpbs.com,2002.

[56] IBM LoadLeveler User’s Guide. IBM Corp., 1993.[57] M. Litzkow, M. Livny, and M. Mutka, “Condor—A Hunter of Idle

Workstations,” Proc. Eighth Int’l Conf. Distributed ComputingSystems, pp. 104-111, June 1988.

[58] I. Foster, C. Kesselman, J. Tedesco, and S. Tuecke, “GASS: A DataMovement and Access Service for Wide Area ComputingSystems,” Proc. Sixth Workshop I/O in Parallel and DistributedSystems, May 1999.

[59] W. Allcock, J. Bester, J. Bresnahan, A. Chervenak, L. Liming, andS. Tuecke, “GridFTP: Protocol Extension to FTP for the Grid,”Grid Forum Internet-Draft, Mar. 2001.

[60] The storage resource broker, http://www.npaci.edu/dice/srb,2002.

[61] H. Casanova, T. Bartol, J. Stiles, and F. Berman, “DistributingMCell Simulations on the Grid,” Int’l J. High PerformanceComputing Applications, vol. 14, no. 3, pp. 243-257, 2001.

[62] H. Casanova and F. Berman, Parameter Sweeps on the Grid withAPST, chapter 33. Wiley Publishers, Inc., 2002.

[63] Sun Microsystems Grid Engine, http://www.sun.com/gridware/,2002.

[64] Entropia Inc., http://www.entropia.com, entropia, year?[65] United Device Inc., http://www.ud.com, 2002.[66] J. Linderoth, S. Kilkarni, J.-P. Goux, and M. Yoder, “An Enabling

Framework for Master-Worker Applications on the Computa-tional Grid,” Proc. Ninth IEEE Symp. High Performance DistributedComputing, pp. 43-50, Aug. 2000.

[67] E. Heymann, M. Senar, E. Luque, and M. Livny, “AdaptiveScheduling for Master-Worker Applications on the ComputationalGrid,” Proc. IEEE/ACM Int’l Workhop Grid Computing (GRID 2000),Dec. 2000.

[68] G. Shao and R. Wolski, and F. Berman, “Master/Slave Computingon the Grid,” Proc. Ninth Heterogeneous Computing Workshop, pp. 3-16, May 2000.


[69] P. Tang and P.-C. Yew, “Processor Self-Scheduling for MultipleNested Parallel Loops,” Proc. 1986 Int’l Conf. Parallel Processing,pp. 528-535, Aug. 1986.

[70] C.P. Kruskal and A. Weiss, “Allocating Independent Subtasks onParallel Processors,” IEEE Trans. Software Eng., vol. 11, no. 10,pp. 1001-1016, Oct. 1985.

[71] C.D. Polychronopoulos and D.J. Kuck, “Guided Self-Scheduling:A Practical Scheduling Scheme for Parallel Supercomputers,”IEEE Trans. Computers, vol. 36, no. 12, pp. 1425-1439, Dec. 1987.

[72] T.H. Tzen and L.M. Ni, “Trapezoidal Self-Scheduling: A PracticalScheme for Parallel Compilers,” IEEE Trans. Parallel and Dis-tributed Systems, vol. 4, no. 1, pp. 87-98, Jan. 1993.

[73] S. Flynn Hummel, J. Schmidt, R.N. Uma, and J. Wein, “Load-Sharing in Heterogeneous Systems via Weighted Factoring,” Proc.Eighth Ann. ACM Symp. Parallel Algorithms and Architectures,pp. 318-328, June 1996.

[74] D.H. Bailey, E. Barszcz, J.T. Barton, D.S. Browning, R.L. Carter, L.Dagum, R.A. Fatoohi, P.O. Frederickson, T.A. Lasinski, R.S.Schreiber, H.D. Simon, V. Venkatakrishnan, and S.K. Weeratunga,“The NAS Parallel Benchmarks,” Int’l J. Supercomputer Applica-tions, vol. 5, no. 3, pp. 63-73, 1991.

[75] “Persistence of Vision Raytracer,” Persistence of Vision Develop-ment Team, 1999.

[76] L.F. Ten Eyck, J. Mandell, V.A. Roberts, and M.E. Pique,“Surveying Molecular Interactions with DOT,” Proc. 1995 ACM/IEEE Supercomputing Conf., pp. 506-517, Dec. 1995.

[77] W. Cirne and F. Berman, “Using Moldability to Improve thePerformance of Supercomputer Jobs,” J. Parallel and DistributedComputing, vol. 62, no. 10, pp. 1571-1601, 2002.

[78] W. Cirne and F. Berman, “When the Herd Is Smart: The AggregateBehavior in the Selection of Job Request,” IEEE Trans. Parallel andDistributed Systems, vol. 14, no. 2, pp. 181-192, Feb. 2003.

[79] “Extensible Argonne Scheduler System (EASY),”http://info.mcs.anl.gov/Projects/sp/scheduler/scheduler.html, 2002.

[80] “Maui Scheduler,” http://supercluster.org/maui/, 2002.[81] “Load Sharing Facility (LSF),”http://wwwinfo.cern.ch/pdp/lsf/,

2002.[82] W. Cirne and F. Berman, “A Model for Moldable Supercomputer

Jobs,” Proc. IPDPS 2001—Int’l Parallel and Distributed ProcessingSymp., Apr. 2001.

[83] D.G. Feitelson, “Metrics for Parallel Job Scheduling and TheirConvergence,” Job Scheduling Strategies for Parallel Processing,vol. 2221, pp. 188-206, 2001.

[84] J. Gehring and A. Reinefeld, “MARS—A Framework for Mini-mising the Job Execution Time in a Metacomputing Environ-ment,” Future Generation Computer Systems, vol. 12, pp. 87-99, 1996.

[85] H. Topcuoglu, S. Hariri, W. Furmanski, J. Valente, I. Ra, D. Kim, Y.Kim, X. Bing, and B. Ye, “The Software Architecture of a VirtualDistributed Computing Environment,” Proc. Sixth IEEE High-Performance and Distributed Computing Conf. (HPDC ’97), pp. 40-49,1997.

[86] M. Sirbu and D. Marinescu, “A Scheduling Expert Advisor forHeterogeneous Environments,” Proc. Heterogeneous ComputingWorkshop (HCW ’97), pp. 74-87, 1997.

[87] J. Budenske, R. Ramanujan, and H.J. Siegel, “On-Line Use of Off-Line Derived Mappings for Iterative Automatic Target Recogni-tion Tasks and a Particular Class of Hardware,” Proc. Hetero-geneous Computing Workshop (HCW ’97), pp. 96-110, 1997.

[88] P. Au, J. Darlington, M. Ghanem, Y. Guo, H. To, and J. Yang,“Co-Ordinating Heterogeneous Parallel Computation,” Proc.Euro-Par ’96, pp. 601-614, 1996.

[89] J. Arabe, A. Beguelin, B. Lowekamp, E. Seligman, M. Starkey, andP. Stephan, “ Dome: Parallel Programming in a HeterogeneousMulti-User Environment,” Proc. 10th Int’l Parallel Processing Symp.,pp. 218-224, 1996.

[90] J. Gehring, “Dynamic Program Description as a Basis for RuntimeOptimisation,” Proc. Third Int’l Euro-Par Conf., pp. 958-965, Aug.1997.

[91] J. Schopf, “Performance Prediction and Scheduling for ParallelApplications on Multiuser Clusters,” Ph.D. thesis, Univ. ofCalifornia, San Diego, Dec. 1998

[92] GrADS project homepage, http://nhse2.cs.rice.edu/grads, 2002.[93] Grid research and innotation laboratory homepage, http://

grail.sdsc.edu, 2002.[94] Virtual Instrument project, http://gcl.ucsd.edu/vi_itr/, 2002.

[95] M. Faerman, A. Birnbaum, H. Casanova, and F. Berman,“Resource Allocation for Steerable Parallel Parameter Searches,”Proc. Grid Computing Workshop, Nov. 2002.

[96] D. Kondo, H. Casanova, E. Wing, and F. Berman, “Models andScheduling Mechanisms for Global Computing Applications,”Proc. Int’l Parallel and Distributed Processing Symp. (IPDPS), Apr.2002.

[97] I. Foster, C. Kesselman, and S. Tuecke, “The Anatomy of the Grid:Enabling Scalable Virtual Organizations,” Int’l J. High PerformanceComputing Applications, 2001.

Francine Berman received the BA degree fromthe University of California, Los Angeles, in1973, the MS degree from the University ofWashington in 1976, and the PhD degree fromthe University of Washington in 1979. She is thedirector of the San Diego Supercomputer Centerand the National Partnership for AdvancedComputational Infrastructure, a fellow of theACM, a professor of computer science andengineering at the University of California, San

Diego, and founder of the UCSD Grid Laboratory. Over the last twodecades, her research has focused on the development of software,tools, and models for Parallel and, more recently, Grid Computingenvironments. Dr. Berman has served on numerous program commit-tees, advisory boards, and steering committees and currently serves asone of the two principal investigators for the US National ScienceFoundation’s TeraGrid project and as co-PI on the Extensible TerascaleFacility (ETF) project. She is a member of the IEEE Computer Society.

Richard Wolski received the BS degree incomputer science from the California Polytech-nic State University at San Luis Obispo, and theMS and PhD degrees from the University ofCalifornia Davis, Lawrence Livermore NationalLaboratory (LLNL) campus. He is an assistantprofessor of computer science at the Universityof California, Santa Barbara. His researchinterests include Grid computing, distributedcomputing, scheduling, and resource allocation.

In addition to AppLeS, he leads the Network Weather Service projectwhich focuses on online prediction of resource performance. He hasdeveloped EveryWare—a set of tools for portable Grid programming. Heis also leading the G-commerce project studying computationaleconomies for the Grid. He is a member of the IEEE.

Henri Casanova received the BS degree fromthe Ecole Nationale Superieure d’Electronique,d’Informatique et d’Hydraulique de Toulouse,France, in 1993, the MS from the Universite PaulSabatier, Toulouse, France, in 1994, and thePhD degree from the University of Tennessee,Knoxville, in 1998. He is an adjunct professor ofcomputer science and engineering at the Uni-versity of California, San Diego, a research

scientist at the San Diego Supercomputer Center, and the founder anddirector of the Grid Research And Development Laboratory (GRAIL) atUCSD. His research interests are in the area of parallel, distributed,Grid, and Internet computing, with emphases on modeling, scheduling,and simulation. He is a member of the IEEE.


Walfredo Cirne received the BS and MSdegrees from the Universidade Federal daParaiba and the PhD degree from the Universityof California, San Diego. He is a professor at theUniversidade Federal de Campina Grande,Brazil. He has previously worked in machinelearning and computer networks. Now, hisresearch efforts concentrate in Grid Computingand Service Availability, areas in which he leads

two research projects. MyGrid (http://www.dsc.ufpb.br/mygrid/) aims toprovide user-level support for light parallel applications running oncomputational grids. Smart Alarms (http://www.dsc.ufpb.br/~smart/)investigates how to determine the root cause of problems in largeelectrical transmission networks based on the sequence of alarmsoriginating from the components of the network.

Holly Dail received the BS degree in physicsand in oceanograpny from the University ofWashington and the MS degree in computerscience from the University of California, SanDiego. She is a research programmer at the SanDiego Supercomputer Center. Her current re-search interest is in application developmentenvironments for Grid computing. She is amember of the IEEE Computer Society.

Marcio Faerman received the MS and BSdegrees in electrical engineering from Universi-dade Federal do Rio de Janeiro. He is a PhDcandidate in the Department of ComputerScience and Engineering of University of Cali-fornia, San Diego. His research interests includegrid computing, application-level scheduling,computer networks, performance modeling,and prediction. He is a student member of theIEEE and the IEEE Computer Society.

Silvia Figueira received the BS and MSdegrees in computer science from the FederalUniversity of Rio de Janeiro (UFRJ), Brazil, in1988 and 1991, respectively, and the PhDdegree in computer science from the Universityof California, San Diego, in 1997. She wasinvolved in research as a member of thetechnical staff of NCE/UFRJ (Computing Centerof UFRJ) from 1985 to 1991. Currently, she is anassistant professor of computer engineering at

Santa Clara University. Her current research interests are in the area ofsystem support for parallel computation in network of workstations andGrid environments.

Jim Hayes received the BS degree in informa-tion and computer science from the University ofCalifornia, Irvine, and the MS degree in compu-ter science from the University of California, SanDiego. He is currently a research programmer atthe San Diego Supercomputer Center. Hisparticular interest lies in incorporating flexibilityinto the design of Grid software and systems toallow for later modification as requirementschange. His primary projects involve programs

and libraries that speed the process of adapting computational sciencecodes on Grid resources.

Graziano Obertelli received the Laurea degreefrom the Dipartimento di Scienze dell’Informa-zione at the Universita di Milano. He is currentlya research programmer in the Department ofComputer Science at the University of California,Santa Barbara. His research interests are in thearea of Grid computing.

Jennifer Schopf received the BA degree fromVassar College and the MS and PhD degreesfrom the University of California, San Diego. Sheis an assistant computer scientist at the Dis-tributed Systems Lab, part of the Mathematicsand Computer Science Division at ArgonneNational Lab. Her research is in the area ofmonitoring, performance prediction, and re-source scheduling and selection. She is cur-rently part of the Globus Project and is on the

steering group of the Global Grid Forum as the area codirector forScheduling and Resource Management. She is a member of the IEEEand the IEEE Computer Society.

Gary Shao recieved the BS degree in electricalengineering and the BS degree in computingengineering from the University of Missouri-Columbia in 1982, the MS degree in electricalengineering from Washington University, St.Louis in 1992, and the PhD degree in computerscience from the University of California, SanDiego, in 2001. His current research interest are

in Grid computing and network modeling.

Shava Smallen received the BS and MSdegrees from the Department of ComputerScience and Engineering at the University ofCalifornia, San Diego, in 1998 and 2001. She isa research programmer at the San DiegoSupercomputer Center. Her current researchinterest is in Grid computing.

Neil Spring received the BS degree in computerengineering from the University of California, SanDiego in 1997 and the MS degree in computerscience from the University of Washington in2000. He is a PhD student at the University ofWashington. His research interests include net-work topology discovery, congestion control,network performance analysis, distributed oper-ating systems, adaptive scheduling of distributed

applications, and operating system support for networking.

Alan Su received the BS degree in electricalengineering and computer science from theUniversity of California, Berkeley. He is a PhDcandidate in the Department of ComputerScience and Engineering at the University ofCalifornia, San Diego. His research interests lieprimarily in the area of scheduling problems forcomputational Grid environments. In particular,he is focusing on the challenges associated withparallelizing entity-level scientific applications.

Dmitrii Zagorodnov received the BS and MSdegrees in computer science from the Universityof Alaska, Fairbanks, in 1995 and 1997, respec-tively. Since September 1997, he has beenworking toward the PhD degree in computerscience at the University of California, SanDiego. His research interests include distributedoperating systems, with emphasis on fault-tolerant network services.

. For more information on this or any other computing topic,please visit our Digital Library at http://computer.org/publications/dlib.


Documents

Adaptive computing on the grid using AppLeS - Parallel and ... · 2APPLES: PRINCIPLES AND FUNDAMENTAL CONCEPTS Initiated in 1996 [2], the goals of the AppLeS project have been twofold