Embed Size (px)
Citation preview
J Grid Computing (2011) 9:355–377DOI 10.1007/s10723-011-9192-1
Infrastructure Federation Through Virtualized Delegationof Resources and ServicesDGSI: Adding Interoperability to DCI Meta Schedulers
Georg Birkenheuer · André Brinkmann ·Mikael Högqvist · Alexander Papaspyrou ·Bernhard Schott · Dietmar Sommerfeld ·Wolfgang Ziegler
Received: 28 February 2010 / Accepted: 12 June 2011 / Published online: 7 July 2011© Springer Science+Business Media B.V. 2011
Abstract Infrastructure federation is becomingan increasingly important issue for modern
This work is supported by the German FederalMinistry of Education and Research underproject grant #01IG09009.
G. Birkenheuer · A. BrinkmannPaderborn Center for Parallel Computing,University of Paderborn, Paderborn, Germany
G. Birkenheuere-mail: [email protected]
A. Brinkmanne-mail: [email protected]
M. HögqvistZuse Institute Berlin, Berlin, Germanye-mail: [email protected]
A. Papaspyrou (B)Technische Universität Dortmund,Dortmund, Germanye-mail: [email protected]
B. SchottPlatform Computing GmbH, Ratingen, Germanye-mail: [email protected]
D. SommerfeldGesellschaft für wissenschaftliche DatenverarbeitungmbH Göttingen, Göttingen, Germanye-mail: [email protected]
W. ZieglerFraunhofer Institute SCAI, Sankt Augustin, Germanye-mail: [email protected]
Distributed Computing Infrastructures (DCIs):Dynamic elasticity of quasi-static Grid environ-ments, incorporation of special-purpose resourcesinto commoditized Cloud infrastructures, cross-community collaboration for increasingly diverg-ing areas of modern e-Science, and Cloud Burst-ing pose major challenges on the technical levelfor many resource and middleware providers. Es-pecially with respect to increasing costs of oper-ating data centers, the intelligent yet automatedand secure sharing of resources is a key factorfor success. With the D-Grid Scheduler Inter-operability (DGSI) project within the GermanD-Grid Initiative, we provide a strategic tech-nology for the automatically negotiated, SLA-secured, dynamically provisioned federation ofresources and services for Grid-and Cloud-typeinfrastructures. This goal is achieved by comple-menting current DCI schedulers with the ability tofederate infrastructure for the temporary leasingof resources and rechanneling of workloads. Inthis work, we describe the overall architectureand SLA-secured negotiation protocols withinDGSI and depict an advanced mechanism for re-source delegation through means of dynamicallyprovisioned, virtualized middleware. Throughthis methodology, we provide the technologicalfoundation for intelligent capacity planning andworkload management in a cross-infrastructurefashion.
356 G. Birkenheuer et al.
Keywords Grid computing · Cloud computing ·Resource management · Negotiation ·Virtualization · Middleware · SLAs
1 Introduction
Grid computing technology has reached a stateof maturity where former enthusiastic users havebecome customers, and “nice-to-have” resourcesof (mostly) academic origin, usually occupied withcutting-edge research problems, start to providevalue to projects driven by the needs of the verybusinesses they comprise.
A good example is climate research: Starting toexplore Grid infrastructures as a new way to copewith tera-scale1 data sets, the climate communitynow sees its portal-based gateways to resourcesas a tool to provide crucial information to mem-bers of parliament in the European member statesand to decision makers from major reinsurancecompanies. Another case is provided by plasmatechnology consultancy: Within a business ecosys-tem of Small and Medium Enterprises (SME)in mechanical engineering, physicist modelers ofplasma gas behavior, and consultancy bureaus,Grid technology is employed in a very similar way.Customers that build new machinery for makingnext-generation material science through plasmacoating happen to give their blueprints to highlyspecialized consultants who—using Grid technol-ogy for highly compute-intensive simulations—answer questions regarding the feasibility of cer-tain configurations by relying upon the subtlephysical interaction models only specialist physi-cists can provide.
Overall, the usage of Grid beyond research isstarting to plant its roots. Of course, this new cus-tomer base has different requirements for a Grid:Aspects such as resource reliability, performanceguarantees, and data secrecy become more impor-tant than they have been in classic Grid use casesthat stem from academia. The IntergovernmentalPanel of Climate Change (IPCC), for example,
1Terabytes of data per input set for a single task in acomplex workflow.
demands results on a certain quality level, andplasma consultants have tight deadlines for theirreports. As such, both projects start to look outfor a more flexible approach to reliably integrateadditional resources at times of demand.
Besides Grid computing, the more recent con-cept of Cloud Computing has reached the mar-ket, beginning to deliver market value and gen-erating revenue in the large scale. The naturalidea for the aforementioned use cases wouldthus be to embrace Infrastructure as a Service(IaaS) concepts for their natural purposes. Still,the precious compute and storage infrastructureand community-specific high level services fromthe Grid era (think intelligent special-purposedata management and highly customized por-tal applications) that are successfully used by alarge community cannot be sacrificed. In addi-tion, there is a strong need to keep the col-laborative VO-based approach: Partners, albeitcompeting at the market, want to collaborateover the common platform. Discussions with thedifferent stakeholders elicited the wish for conver-gence between the traditional Grid approach, withmore or less static infrastructure environmentsand value-added community services, and theelastic yet “baseline services only”-enabled Cloudapproach.
Through the use of open standards, the D-Grid Scheduler Interoperability (DGSI) projectaims to make this convergence happen on thetechnology level. Initially started as an effort tomake cross-VO collaboration between ServiceGrids possible and to connect researchers fromdifferent communities on the infrastructure level,the project extended its goals towards linking tra-ditional Grid environments with modern elasticinfrastructures. The support for auto-negotiated“leasing” of resources and delegation of work-loads between different types of infrastructuresopens new opportunities of cross-community andcross-infrastructure interoperation, and strict end-to-end Service Level Agreements ensure theapplicability to business-driven Grid communi-ties. This federation of different paradigms forresource exposure forms a key issue in theEuropean resource space. Open standards, espe-cially from organizational bodies that think into
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 357
both Grid and Cloud directions, build the techno-logical foundation for this challenge. By adoptingthese standards for both academic and commer-cial environments, collating them according to therequirements of business-driven Grid infrastruc-tures and contributing back to the standards or-ganizations, DGSI provides a strategic technologyplatform for the future convergence of Grid andCloud computing.
In this paper, we describe the DGSI architec-ture, protocols, and technologies and detail twoapproaches for DCI federation: In resource del-egation, a meta scheduler can place a resourceavailable within its own domain under the exclu-sive planning authority of another meta scheduleroutside of his administrative control, for a previ-ously negotiated time window. In activity delega-tion, one meta scheduler hands hands an activityand the management of its execution over to themanagement domain of another scheduler. Bothapproaches aim to work in a federated, cross-community, cross-infrastructure scenario.
The remainder of the paper is organized asfollows: In Section 2, we give an overview ofrelated work with respect to capacity planningin federated DCI infrastructures and show var-ious projects employing negotiation and agree-ment mechanisms for workload management. InSection 3, we give an overview of the DGSIarchitecture, providing details on the two dele-gation approaches. In Section 4, we discuss thenegotiation and agreement details with respectto the DGSI protocol and show the integrationof open standards within the whole system. InSection 5, we show the details of the most ad-vanced approach, namely virtualized resource del-egation and discuss technological challenges inrealization. In Section 6, we detail the informationmodel used in DGSI. In Section 7, we evaluateour approach using a test environment. Finally, inSection 8, we conclude our work with prospectsfor future development.
2 Related Work
Federation aspects in general have been discussedmostly from a security perspective [11], on the
level of networks [33], or for monitoring pur-poses [7]. Although standardization is considereda key case [15] and has been shown on an oper-ational level [32], federation protocols for auto-matic, infrastructure-level, and SLA-secured pro-visioning of resources and services are yet to beshown.
This notion of federating Grid and Cloud in-frastructures on the broker level touches twomajor topics addressed by research in the lastdecade: scheduling and planning federated DCIinfrastructures and interoperation protocols fornegotiation and agreement between indepen-dently acting brokers in the domain of resourcemanagement.
2.1 Capacity Planning in ModernDCI Environments
Automated workload scheduling in distributedcomputing infrastructure (DCI) systems is a well-covered research topic and stems from classicparallel machine scheduling problems. The fed-erated nature of such systems, however, allowsa much more efficient usage of resources dueto the shared utilization by a large user commu-nity [5].
During the last years, a remarkable amountof effort has been put into workload delegationwithin federated DCI ecosystems, mostly withinthe broader context of Grid computing. Onepopular approach in this scenario is to assumecentralized scheduling services [18]. For example,Ernemann et al. [13] show advantages of hier-archical scheduling in general by considering theAverage Weighted Response (AWRT) objective.Furthermore, Kurowski et al. [25] identify multi-ple objectives for efficient job scheduling in Gridsand propose a strategy based on prediction mech-anisms and resource reservation. For decentral-ized environments, only few results that supportthe delegation of workload have been published.England and Weissman [12] give an estimationof load sharing costs and benefits relying on syn-thetic workloads only. Grimme et al. [20] ana-lyze the prospects of collaborative job sharingand compare their results to the non-cooperativescenario of the same machines. Recent work of
358 G. Birkenheuer et al.
Fölling et al. [16, 17] proposes a fuzzy-based,evolutionary-optimized exchange policy for a fullydecentralized scenario, which shows robustnesseven in changing environments and automaticallyadapts to the current local load.
With the elasticity of IaaS-supported DCI en-vironments, a new kind of flexibility challengescurrent scheduling approaches due to the inherentreconfigurability of machines and the resultingchanges in scheduling responsibilities. Up to now,this aspect has only occasionally been discussed inresearch: Since the complexity of operating suchsystems in the large scale and the feasibility ofprovisioning and reconfiguring them on demandhampered the realization for production envi-ronments, discussion focused on rather low-levelcomputer hardware. For example, Kota et al. [24]consider the problem of scheduling and mappingof tasks onto reconfigurable logic units for a givenapplication introducing a concept of parameter-ized modules. Their approach is a typical exam-ple of scheduling in the context of reconfigurablehardware that involves varying sizes of availablehardware. Subramaniyan et al. [39] transferredsimilar ideas to the HPC context and analyzedthe dynamic scheduling of large-scale HPC ap-plications in parallel reconfigurable computingenvironments. They assess the performance ofseveral common HPC scheduling heuristics thatcan be used by an automated job managementservice to schedule application tasks on parallelreconfigurable systems. However, their approachis limited to a single HPC system and does notinvolve the interaction of multiple autonomouspartners in a DCI environment.
Iosup et al. have introduced the concept ofdelegated matchmaking, where resources fromremote Condor pools can be bound temporarilyto the local environment [22]. The solution aug-ments a hierarchy of Grid sites with peer-to-peerconnections between sites on the same hierarchylevel. The basic idea of the approach is to com-bine advantages of hierarchical and distributedapproaches to manage interoperating Grids. Themain aim of the work is to decrease the necessaryadministrative effort by delegating resources tojobs instead of moving jobs to resources.
All approaches, however, share a strong focuson algorithmic improvements and assume an ap-propriate delegation infrastructure to be in place.
2.2 Negotiation and Agreement for ResourceManagement
During the last couple of years, the interest inusing negotiation during job submission increasedconstantly. We identify two main reasons for this:
1. With Grid and Cloud infrastructures becom-ing an accepted way to extend the local re-sources, job submission may include resourcesbeyond the own administrative domain. As aconsequence, Quality of Service (QoS) of a re-mote submission cannot be enforced throughlocal policies but has to be agreed upon withthe external service provider, i.e., an agentlike a meta-scheduler acting on behalf of thisservice provider.
2. Standards for creating such agreements,namely GLUE [2] for a description of re-sources, JSDL [4] for describing the propertiesof a job, and WS-Agreement [3] for negoti-ating and creating the agreement on resourceusage and the related QoS, have been adoptedand thus become available.
This growing interest is reflected by the increasingnumber of projects and developments indifferent areas implementing negotiation forjob submission. Two surveys in 2007 [30, 35]identified more than 10 projects by that timeeither already providing an implementation,working on an implementation or planning toimplement negotiation in existing systems, e.g.,VIOLA MetaScheduling Service, AssessGridCCS, ASKALON, Community SchedulingFramework (CSF), AgentScape, CATNETS,Job Submission Service (JSS), GRMS, GridWay,eNanos, and Grid Superscalar.
More recently, a number of projects or devel-opments have been launched or completed thataddress the negotiation and creation of ServiceLevel Agreements for resource management atdifferent levels:
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 359
PHOSPHORUS uses negotiation and ServiceLevel Agreements to co-allocate computationalresources and network resources for submis-sions of jobs, which require a dedicated QoS forboth of these resources [31].
BREIN addresses outsourcing to increase thecompetitiveness of SMEs by enhancing theclassical Grid solutions through integration ofmulti-agent and semantic web concepts intoa dynamic, standards-based environment foreBusiness. The main emphasis has been onmoving away from the Grid approach of han-dling individual resources, to a framework,allows providing and selling services whichrepresent a combination of different resourcetypes [10].
SmartLM focuses its research on the negotiationof the co-allocation of software licenses andGrid resources with respect to job submissionand execution for license-protected applica-tions in a Grid environment [37].
Bazaar is a tool aiming to make the processof SLA-based resource allocation manageableand transparent. The SLA layer is used tofilter out appropriate resources for a job orto identify missing resources. Bazaar is imple-mented in the Central European EGEE-IIIregion [8].
SORMA addresses job submission on a more ab-stract level. It is a research project investigatingthe practical application of computational andother related resources in a market infrastruc-ture. A key aspect of this approach is the in-troduction of potential consumers to fitting re-source providers, which is facilitated throughasynchronous auctions, where “best” fits areidentified and introduced [38].
SLA@SOI addresses the entire frameworkfor SLAs in Service Oriented Infrastruc-tures (SOI). The project focuses on researchand development of the entire architecture forSLAs in SOIs including the provision of theoverall technical framework and the businessrequirements [36].
Although the usage of negotiation and agree-ment for the management of workload has been
adopted by many research efforts, the notionof employing such mechanisms for infrastructurefederation has yet to be tackled.
3 Overview of the Federation Architectureof DGSI
Most DCI environments share the ability toefficiently distribute user workload to the re-sources available. This issue, usually generalizedunder the term Meta Scheduling, is already verydiverse within a community: Both submitted jobsand available resources differ considerably, to theextent that coordination has to handle specializedknowledge about usage scenarios and infrastruc-ture. This leads to very different, community-specific approaches for the development of Gridscheduling services.
The D-Grid Scheduler Interoperability (DGSI)provides a standards-based interoperability layerfor scheduling and resource management servicesin DCI ecosystems. The DGSI solution allows theusers of a community to distribute the workloadamong resources within the management domainof another community. It offers new perspectivesfor community collaboration, resource federation,and efficient utilization.
A general overview of the DGSI architectureis given in Fig. 1. For this approach, we foreseetwo scenarios to be realized within the framework:the delegation of activities and the delegation ofresources.
3.1 Delegation of Activities
By means of DGSI, meta-schedulers are able toexchange activities, i.e., single or parallel jobs orworkflows. In a first step, each meta-schedulerregisters at a distributed registry and publishesits resources and execution capabilities. Now, an-other registered scheduler is able to query for thecapabilities that are requested by an activity itwants to delegate. Furthermore, a session objectwith limited lifetime is generated, which providesa unique interface to a single or multiple activitiesand assures a thread safe access to them. This
360 G. Birkenheuer et al.
Fig. 1 Overall delegationarchitecture using theDGSI protocols. Metaschedulers from differentdomains (architectural,organizational, andtechnological) cooperateusing activity andresource delegation
Delegating Community Scheduler
Community scheduler
Middleware
Agreement
Middleware
LRMS
Proxy
LRMS
Delegate Activity
Ressouce Delegation
Resources free Need for additionalresources
session object containing activities is offered toone or more meta-schedulers that match the de-mands. Every queried meta-scheduler is inspectsthe session and its containing activities indepen-dently and decides whether to accept or denythem. This phase could contain different nego-tiation protocols implementing different behav-ior (Take it or leave it, Renegotiation by use ofcounter-of fers etc.). Eventually, one distinct meta-scheduler—either a local or a remote one—willexecute the activity on its local resources.
3.2 Delegation of Resources
The method of resource delegation has becomean interesting approach due to its advantages,like support of local requirements, e.g., spe-cial workflow scheduling tools or better controlover the delegated resources. Resource delegationaims to
1. Encapsulation of the delegated resources,2. Monitoring of the agreed terms of usage , and3. Hiding of the security-relevant adaptations
and of the delegation of rights.
The DGSI project has identified two distincttechnologies for resource delegation.
The first realization is based on a proxy ap-proach, which focuses on the management ofdelegated resources and is exclusively realizedon the layer of the base middleware. The com-munication between Grid scheduler and mid-dleware is directed over the proxy which en-forces the compliance with the negotiated SLA.In this approach the provider keeps full controlover the infrastructure. No modifications are re-quired for the existing middleware and LRMSsystems.
The second realization is based on virtualiza-tion. This approach allows the configuration of thebase middleware and lightweight local RMS sys-tem in a virtual machine. The approach is flexiblesince the resources can be prepared for every mid-dleware or LRMS. Access rights for the resourcescan be confi gured as part of the deployment.
While both delegation approaches are going tobe implemented within the project, in this paperwe will only focus on architectural aspects of thelatter and detail the technical issues of the mostadvanced realization, namely the virtualization
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 361
approach. To this end, we first introduce the basicnegotiation and agreement mechanisms and theirtechnical rendering.
4 Protocol Definition for Negotiation-basedDelegation of Services
The requirements for the architecture and theprotocol language for delegation were dis-cussed [21] in the Open Grid Forum’s GridScheduling Architecture Research Group (GSA-RG) for some time. The results of this work, alongwith the results and experiences from a numberof projects (see Section 2.2) strongly suggest twomajor design decisions for managing delegations:
1. An agreement on resource delegation canonly be achieved by a negotiation process. Thedifferent administrative domains that make upa Grid or a Cloud infrastructure simply pro-hibit enforcement of remote resource usage.
2. Use of standards as far as possible to increaseinteroperability and usability.
The DGSI approach is thus based on three rel-evant standards to realize its delegation protocol:
– WS-Agreement to negotiate Service LevelAgreements for using remote resources, fordelegating activities to them or for temporarylogical inclusion into the local resource pool.
– Job Submission Description Language(JSDL) to describe requirements for thedelegation. Use case elicitation has shown thatJSDL is suitable to describe the requirementsfor both resource and activity delegation(using only a subset of the language).
– GLUE to describe resources, which will becontributed for resource delegation.
The following sections present WS-Agreementand JSDL, the rationales for using them and de-scribe their use and implementation in the DGSIapproach. Although GLUE plays an importantrole in the protocol definition, it merely providesthe information modeling part for describing re-sources. As such, it is described separately inSection 6.
4.1 WS-Agreement
WS-Agreement is a proposed recommendationof the Open Grid Forum defined by the GridResource Allocation Agreement Protocol work-ing group (GRAAP-WG) [19]. The objectiveof the WS-Agreement specification is to definea language and a protocol for advertising thecapabilities of service providers and creatingagreements based on templates and for mon-itoring agreement compliance at runtime. Anagreement between a service consumer and aservice provider specifies one or more ser-vice level objectives as expressions of the ser-vice consumer’s requirements and the serviceproviderÅRs assurances on the availability of re-sources and/or service qualities. An agreement lifecycle includes creation and termination and moni-toring of agreement states. For example, an agree-ment may provide assurances on the bounds ofservice response time and service availability. Al-ternatively, it may provide assurances on the avail-ability of minimum resources such as memory,CPU MIPS, storage, or a software license.
4.1.1 Rationale
We selected WS-Agreement to implement the ba-sic protocol for resource delegation for the follow-ing reasons:
1. Delegation of resources or activities across au-tonomous communities can only be achievedthrough negotiation of the delegation betweenthe requesting and the providing communityGrid-scheduler.
2. For a negotiation it is essential that a pro-tocol exists, which can be followed, and thatthe two negotiating parties have a commonunderstanding of the objects the negotiationis about. Both can be achieved with WS-Agreement through its protocol and the use ofagreed term languages (like JSDL) to describethe objects.
3. The negotiations should result in ServiceLevel Agreements with binding character forboth parties to deliver a reliable service tothe end-user. Such SLAs are the outcome ofsuccessful negotiations with WS-Agreement.
362 G. Birkenheuer et al.
4. The D-Grid infrastructure is currently beingextended with a layer for the management ofService Level Agreements. This layer will beusing WS-Agreement for the SLAs.
In addition to this, WS-Agreement enjoys wideadoption in the context of DCI environments be-cause of two important properties.
WS-Agreement is standard. WS-Agreement is theonly open standard specifying a language anda protocol for creating Service Level Agree-ments. Besides WS-Agreement only propri-etary solutions are available, most of them de-veloped in and for a certain ecosystem andno longer maintained or further developed,e.g,. the NextGRID SLA [27], or availableunder a commercial license only, like IBM’sWSLA [41]. WS-Agreement uses the Web Ser-vices Resource Framework (WSRF) [42], anopen framework for modeling and accessingstateful resources using Web services, whichhas been published as a standard by the OASISconsortium. The current version of UNICOREand the most popular edition of Globus Toolkitemploy WSRF as a key technology; thus, basicinteroperability for accessing stateful resourcesis already provided by the middleware.
WS-Agreement is extensible. The intrinsic exten-sibility of WS-Agreement is an important fea-ture when using it as the general technology forcreating SLAs in the heterogeneous environ-ment of D-Grid comprising 20+ communitieswith different resources. Since the specificationis completely neutral with respect to specificterm languages to express Service DescriptionTerms (SDT) and Service Level Objectives(SLO), DGSI can select the best suited de-scription languages to be used for resourcedelegation and plug them into WS-Agreement.This mechanism allows providing a commonapproach for creating agreements on delega-tion while supporting a wide range of het-erogeneous resources and activities. While thecurrent specification of WS-Agreement onlyprovides a basic, single round mechanism fornegotiating an agreement, developments areunder way at OGF to extend the negotiationcapabilities towards multi round negotiationsfor creating an SLA and re-negotiation of SLAs
already in force. A first implementation of WS-Agreement with extensions for negotiations hasbeen realised in the SmartLM project [34]. Infact, we already started integrating the draft forthe WS-Agreement Negotiation specificationfor the second phase of the DGSI project.
4.1.2 Implementation
As mentioned before, DGSI will rely on the SLAmanagement layer of D-Grid, which will baseon, extend, and port existing implementations ofWS-Agreement to benefit from previous expe-riences, e.g., described in [6]. One of the mostcomplete and advanced implementations of theWS-Agreement specification is WS-Agreementfor Java (WSAG4J) [40], which provides an easyto use framework to create, monitor, and termi-nate service level agreements based on the WS-Agreement specification. To accomplish this goal,the framework implements the basic features ofthe WS-Agreement protocol and uses a numberof standards in conjunction with WS-Agreement.Hence, WSAG4J provides a complete develop-ment framework for SLA-based services. DGSIuses features such as the WS-Agreement Agree-mentFactory port type and the WS-AgreementAgreementState port type. It makes use of digi-tal signatures to enforce message integrity (WS-Security) and also provides the listing of existingagreements via WS-Service Groups.
Before executing an agreement, the agreementoffers are validated based on template creationconstraints. The validation is also available dur-ing the negotiation process. Limiting the negoti-ation space with creation constraints defined bythe agreement initiator and/or the agreement re-sponder reduces the complexity of a negotiationand allows to achieve more rapid convergenceof a negotiation towards an agreement. As de-scribed in the WS-Agreement specification, anagreement factory of the resource provider ex-poses an operation for creating an agreement outof an initial set of terms and returns an End-point Reference (EPR) to an Agreement service.The agreement factory of the resource provideralso exposes resource properties like templates—representations of acceptable offers for the cre-ation of an agreement. In DGSI these templates
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 363
contain either the description of resources ac-cepting activity delegation or the descriptions ofresources a provider is willing to temporarily dele-gate. To create an agreement, a client—in DGSI aDCI scheduler—makes an agreement offer basedon a previously retrieved template. Once thetwo parties—the scheduler requesting the dele-gation and the scheduler offering resources fordelegation—agree on the terms of the template,the agreement may be created.
After the creation of the agreement, it willbe monitored to verify whether the agreement isfulfilled or violated. For monitoring the agree-ment, several states are defined in the WS-Agreement specification (see Fig. 2):
– Agreement States corresponding to the agree-ment as a whole,
– Service Term States reflecting the states re-lated to the service terms, and
– Guarantee States indicating the states of theguarantees given for the service terms.
WSAG4J implements these states and pro-vides the interfaces to access them for settingand retrieving. However, it should be noted thatthe information accessible through Service TermStates and Guarantee States is not provided byWS-Agreement internal mechanisms but must begathered by external monitoring services (Fig. 3).
For this purpose the DGSI approach is relyingon external information and monitoring servicesavailable in D-Grid.
4.2 Job Submission Description Language
The description of resources on the negotiationlevel can be realised with JSDL. Basically, wehave one resource-specific JSDL profile. It is usedto describe (a) resources that a provider is will-ing to delegate to another scheduler domain and(b) resources requested by a scheduler aiming totemporarily gain control over remote resources.
4.2.1 Rationale
JSDL was originally developed to describe re-source requirements for job requests. As such,JSDL contains several attributes, which are alsoapplicable for resource delegation requirements.Moreover, JSDL can also be extended usingprofiles that address additional attributes requiredfor resource delegation.
4.2.2 Implementation
In the first phase of DGSI, JSDL is used as itis and will be extended as necessary in the sec-ond phase to accommodate additional require-
Fig. 2 AgreementStates as defined inWS-Agreement,including transitionsto each other. Notethat the Of ferReceivedstate is supplementalto the model
OfferReveived
Pending Observed Complete
Rejected
PendingAndTerminating
ObservedAndTerminating
Terminated
364 G. Birkenheuer et al.
Fig. 3 Transition diagramfor the WS-AgreementService and GuaranteeTerm States
Not Determined
Fulfilled
Violated
(a) Service TermStates.
Not Ready CompletedReady
Processing Idle
(b) Guarantee Term States.
ments. A potential extension for resource del-egation is to support JSDL with a GLUE 2.0extension. This would allow Grid-schedulers tocraft delegation requests containing requirementsfor, e.g., the application environment and accesspolicies.
4.3 Delegation Protocol Steps
To depict the integration of the aforementionedstandards within the DGSI protocol stack, we
show a coarse overview of the necessary steps toperform a delegation (see Fig. 4):
1. The requesting scheduler fetches availableSLA templates from the providing scheduler.These can either describe abstract resources(depicting the general characteristics, that is),which basically serve as job slots for activitydelegation, or concrete machine specificationsfor resource delegation.
2. The providing scheduler returns a set of tem-plates describing services it can offer with re-spect to the current situation. These can now
Fig. 4 Ten steps fornegotiating a delegationwithin the DGSI protocolstack. While activitydelegation only requiressix steps (up to theGetState call foragreement monitoring),resource delegation isbrought to completionwith the tenth step. Note,however, that—from stepsix on—each step isspecific to the concreteusage of the delegatedresource (e.g., a singlejob submission)
Requesting Scheduler Providing Scheduler
1. GetTemplates
AT (JSDL)
O (JSDL)
5a. Accept
6. GetState(Agreement Monitoring)
2.
3.
AgreementStore
GLUE 2.0
8. Submit(JSDL,...)
7. StageIn
10. StageOut
9. Monitoring(Service
Monitoring)4a. Setup/Reservation
Delegation-Access
StoreProxy/
VMMon
ConsumeService
SLA5b. Store
4b. Create/Include
WS-Agreement(SLA Creation)
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 365
be concretized with additional requirementsfrom the requesting scheduler’s side.
3. The requesting scheduler offers the con-cretized template (depicting the aspired SLA)to the providing scheduler.
4. The providing scheduler checks the generalfeasibility and adherence to creation con-straints of the SLA and, if correct, creates acorresponding reservation for the delegation(a); this step can also be performed asynchro-nously. In case of resource delegation, the pro-viding scheduler additionally sets up a GLUEspecification of the delegated resource alongwith the stored agreement.
5. Regardless whether the previous step is donesynchronously or not, the providing scheduleraccepts the agreement and stores it in theagreement store, making it available to bothsigning parties.
6. The providing scheduler gathers necessary in-formation, namely the machine characteristicsfor the delegated system in case of resourcedelegation or the activity state in case of activ-ity delegation. In case of the latter, the step ofthe protocol end here.
7. For resource delegation, the requesting sched-uler is now able to use the delegated resourcelike its own: For potential jobs that are runon the newly available resource, it ensures theavailability of data through regular stage-inmechanisms via the service interface initiallynegotiated and eventually described in theGLUE document.
8. Jobs are regularly submitted to the resourcethat is available, using JSDL described tem-plates.
9. Monitoring of the service is done via a mon-itoring service, exposing the dynamic state ofthe delegated resource.
10. After job completion, data is staged-out again.
5 Virtualization Approach for the Delegationof Resources
While the management of the federated delega-tion ends at the protocol level for activity del-egation, it is obvious that, to ensure delegatedresources to the requesting schedulers, it takes
additional steps on the infrastructure level. Thisdoes not only comprise the architecture itself butalso lower-level details. It is therefore necessaryto detail the scenario within which resource dele-gation takes place and to formulate administrativerequirements to achieve this.
5.1 Scenario of the Resource Delegation
In Cloud computing environments, the delegationof resources is a straightforward process: Giventhe typical IaaS scenario, the customer negotiateswith a Cloud provider over a defined number ofresources. These are usually accessible througha provider-specific interface and get provisionedas virtual machines on top of a hypervisor. Thevirtual machines—containing the software exe-cution environment—are either provided by thecustomer (the prevalent mechanism) or via a soft-ware repository offered by the Cloud provider(providing appliances for certain purposes, whichcan be customized further). Lifecycle manage-ment of the machine and deployment of activitiesor services is completely under the responsibilityof the customer.
This approach is fundamentally different fromthe standard Grid computing scenario: Here, anactivity (usually a job) is given to a Grid provider;lifecycle management of this activity is done bythe provider. This makes the idea of resource del-egation very interesting from a capacity manage-ment point of view: It allows a foreign scheduler,for example, servicing a certain user communitysuch as climate research or plasma physics, toincorporate Grid resources from another sched-uler’s domain to its set of managed resources.
With the DGSI approach, we adopt the ideasand principles from Cloud computing to Gridinfrastructures. Overall, we assume the follow-ing process: Before confirming to an SLA, seeSection 4.3, the providing scheduler (PS) ensuresthrough a Delegation Daemon (DD) running atthe delegating site that the resources are provi-sioned with respect to the Service DescriptionTerms (SDT). The requesting scheduler (RS)is then able to use the delegated resources viaa virtualized, SDT-compliant, on-demand Gridmiddleware (such as Globus GRAM, gLite CE,UNICORE xJS). The SDT-specific middleware
366 G. Birkenheuer et al.
allows the requesting scheduler to use the del-egated resources in the same way as it uses itscommunity resources. Through this virtual front-end (VFE), the RS delivers its workload to theresources.
5.2 Administrative Requirements
Both Grid resources and Grid workload are usu-ally not virtualized. The resulting heterogeneityof the infrastructures is one of the most seriouschallenges of the resource delegation approach.It is therefore necessary to describe additionaladministrative requirements that an infrastructuresuch as DGSI has to comply with in order toallow a delegation of resources within a DCIfederation.
5.2.1 Trust and Security for VirtualFront-End Systems
Security is an important issue for the virtual-ized resource delegation approach as well as forevery other distributed protocol. A main issue isthe responsibility for the security of the virtualmachine images. If the requesting community isresponsible, then why would the providing Gridcommunity trust them? This means that, if a newvirtual Grid middleware is created, this image hasto be certified by a trusted entity and stored on anaccessible server. In the DGSI approach, this is-sue is solved by following the software repositoryapproach from Cloud computing: Middleware im-ages are managed and provisioned by the systemowner, thus allowing him to keep them under hiscontrol.
5.2.2 Firewall
A Grid provider typically has no virtualizationsoftware installed on the computing nodes. There-fore, it is likely that the providing site (i.e., the onethat supports resource delegation) will run oneor more servers that are able to host the VFEs.However, the number of VFEs deployed can vary,and each VFE must (usually because of internalrequirements of the various Grid middlewares)have its own IP and needs connectivity with therequesting schedulers. In the following, we will
assume that there is a known IP range dedicatedto the virtual machines for resource delegation.
Allowing the setup and execution of a virtualmachine to be directly accessible from the Inter-net will not be accepted in most cases. There-fore, it is necessary to have a separate firewallthat only allows connections between the VFEsand their corresponding requesting schedulers.Consequently, the firewall should either allow allconnections on specific ports on the IP range ofthe virtual machines from outside or dynamicallyallow connections from the IP addresses of therequesting schedulers to their assigned VFEs. Inboth cases, the requesting schedulers should re-ceive information about the address of the VFEthat they should connect to.
The latter solution, the more secure one, re-quires that the IP addresses are known, that thefirewall can be updated dynamically, and that theIP address of the requesting scheduler is providedduring negotiation.
5.2.3 Certif ication, Principal, and NamingAdministration
Especially with respect to Grid environments,identity management is a major issue as well. Anew user and certificate might have to be set upfor a machine, and the users and DNs of therequesting community have to be set up by the PS.
Server Certif icates Each VFE needs a certificatefor the virtualization approach. Usually, this doesnot provide a challenge if the resource providingscheduler sets up the VM and can therefore use aprepared certificate matching the Domain NameSystem (DNS) name assigned to the virtual mid-dleware. In case the requesting scheduler providesthe VM image, it is the RS’ task to arrange thecertificate, which is a more dynamic yet elaborateapproach.
Principal Management The providing schedulerhas to handle the user certification. If the cer-tificate of the VFE is provided by the RS, theVFE might be able to load the user DNs viaupdates from a VO Membership RegistrationService (VOMRS) of the RS. This is challeng-ing as the VFE certificate has to be accepted
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 367
by the VOMRS site. However, for many Gridenvironments, this information is not deliveredinstantaneous, but rather scheduled updates (e.g.,once per day) are run, leaving a significant gapin time until the user is known to the under-lying infrastructure. Nevertheless, the resourcesare to be delegated to the RS at once. Supportfor short-lived certificates in this scope mightprovide a solution for this challenge. Interfacingwith the VOMRS would be a possible solutionas well.
5.2.4 Naming Administration
Each VFE has to receive an IP address when it isstarted, and this IP has to be known by the RS.Nevertheless, relying on dynamic DNS services isa challenge as reverse DNS lookups can fail. Aport forwarding solution using a proxy for thispurpose could solve this problem by forwardingconnections from the requesting scheduler to theassigned VFE. The scheduler only needs to know
the address of the proxy and the port to con-nect to.
5.3 Software Architecture
From our example use cases, we assume that theresource delegation is negotiated between twometa Community Grid meta schedulers. After thenegotiation process, the providing scheduler usesthe so-called Delegation Daemon (DD), whichis running at the delegating site and supervisingthe delegation and provisioning. When the DDhas successfully set up a delegation, it creates aGLUE document that contains a detailed descrip-tion of the delegated resources. This documentis linked to the SLA and is accessible throughthe WSAG protocol. The RS can extract all nec-essary information to use the resource from thisdocument.
Overall, the Delegation Daemon poses the coreservice for realizing the delegation of resources,and its components are described in the following.
Fig. 5 Generalarchitecture of theDelegation Daemon(DD), exposing interfacesfor SLA management, filestaging, and VM storage,provisioning, and control
Delegation Daemon
Security Layer
File-StagingDaemon
WSAGInterface
ControlManager
Virt.-Manager
VM-Repository
File Transfers:VM-Images of Node Daemons
Control Delegationby SLA
VMM
368 G. Birkenheuer et al.
For a general overview of the interdependenciesof its various components, see Fig. 5.
5.3.1 The WS-Agreement Interface
The WS-Agreement interface exposes means fordelivering end-to-end SLA capabilities and allowsaccess for managing specific agreements. In par-ticular, it offers services to the PS in order to han-dle delegations. More specifically, the followingmethods are provided:
Management of reservations allows creating,deleting, and modifying reservations on thelower-level infrastructure. Especially for Gridenvironments, this provides unified means formapping Service Terms on the abstract SLAlevel to resource reservations on the concreteLRMS level.
Management of delegations allows creating, delet-ing and modifying delegations on the higherlevel infrastructure.
Monitoring of delegated resources exposes mon-itoring data to the RS or third party servicesin order to assess the service quality providedthrough the delegation and monitor the Guar-antee Terms for their fulfillment.
Information on delegation and reservation pro-vide a GLUE schema-style document, whichdescribes the delegated resources in detail.
5.3.2 The Control Manager
The control manager is responsible for the man-agement of the whole resource delegation processand provides a persistence layer for informationrecovery due to unexpected system outages. Itprovides methods for
Firewall configuration, including management ofrules, on-demand hole punching setup, dynamictunneling for higher-level services, the accep-tance of new VOs, and the setup of users andworkspaces. The control manager has the nec-essary information concerning the delegatedresource after the setup of the delegation hasbeen successfully completed. The control man-ager stores the information in a GLUE docu-
ment and is thus able to access it in the futureto extract all information about this delegation.This document will also be transferred to theRS to transmit all necessary usage information.
Identity management taking care of the accep-tance and incorporation of new VOs, the cre-ation and mapping of new users, and the setupof workspaces;
LRM interfacing performing reservation setup,handling, and enforcement towards the under-lying Local Resource Manager; and
Domain monitoring, including the supervision ofthe firewall (including opened ports) and provi-sioned virtual machines.
In addition, the Control Manager takes care ofVirtualization Manager steering and the creationof the GLUE schema document for a resourcedelegation throughout the setup process.
5.3.3 The Virtualization Manager
The task of the virtualization manager is to start,monitor, and stop the virtual front-ends. A pos-sible option to ensure this would be the manage-ment through state-of-the-art Cloud managementenvironments, such as Eucalyptus [14], Nimbus[28], or OpenNebula [29]. However, the high com-plexity of such tools with many additional soft-ware dependencies outweighs the advantage ofusing already developed software, since a lot offunctionality is deemed unnecessary for the re-source delegation purposes within DGSI. There-fore, it was decided to develop a DGSI-specificvirtualization manager that offers the exact func-tionality needed for resource delegation on thebasis of the widely adopted and de-facto standardlibvirt library [9, 26]. Through this approach,the Virtualization Manager offers functional-ity for
Image copy ensuring the availability of virtualmachine images on the VFE node on-demand;
LRMS adaptation performing reconfiguration ofthe LRM layer with respect to the requesteddelegation;
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 369
Virtual machine setup taking care of the delega-tion instance-specific customization of the VFEvirtual machine;
Monitoring aggregation delivering aggregated in-formation to higher-level services, regardingthe state of the VM system and the LRM.
5.3.4 The Virtual Machine Repository
The VM repository stores the virtual machine im-ages. This enables a requesting scheduler, whichfrequently uses delegated resources, to reuse analready copied virtual machine image. The VMrepository will contain standard Grid middlewareinstallations, including Globus 4.0.x and Globus4.2.x, UNICORE 5 and UNICORE 6.x, and gLite.In addition, the repository can be used to storeadditional software such as the resource manage-ment and scheduling system, which is to be startedon the worker nodes as delegated resource LRMmanagers.
5.3.5 The File Staging Daemon
The file staging daemon allows the RS to transferand store the VM images inside the repository. Al-though being transport-agnostic, we will providean implementation on the basis of GridFTP forthis purpose. A technical issue for the DD is that,due to security requirements, it has to run under aspecific user account, which has to be available onall delegated resources and needs administrativeprivileges to manage the LRMS daemons on thenodes. In modern systems, however, this is nota severe problem: RBAC2 mechanisms ensurethat only certain privileged operations can be is-sued, and impersonation tools like sudo allowthe execution of certain commands as anotheruser without compromising administrative princi-pal account details.
Overall, the delegation process can be depictedas a protocol sequence incorporating the differentservices. In our example (see Fig. 6), the re-questing community chooses the Grid middleware
2Role-Based Access Control
in the virtualization approach. The DD at theprovider starts the image, provides the IP andports accessible in the firewall, and reserves theresources.
5.4 Management of the Virtualized GridMiddleware
The virtualization approach offers a high flexibility as resources can be prepared with an ar-bitrary middleware. Initially, resource providerssupply a set of predefined virtual middleware en-vironments to satisfy common demands. Alterna-tively, the requesting scheduler can be allowedto set up a custom middleware. If the customerhas special demands concerning the delegated re-source, he can submit a specialized middlewareand use the resources through this interface. Thisscenario is relevant if the scheduler needs spe-cial extensions to the middleware or a certainLRMS including workflow orchestration integra-tion. The virtualization approach does not implycode changes in the standard Grid middlewareenvironments. Therefore, if a scheduler does notsupport the interface provided by the proxy ap-proach, it can still use the virtualization approachfor participating in resource delegation.
5.4.1 Realization
The key idea is that the (existing) front-end isresponsible for the start of the virtual front-end(VFE), including the virtual Grid middleware.For this, a delegation daemon with a WSAG in-terface extends the front-end. After the negotia-tion between the Grid schedulers, the providingGrid scheduler hands over the resulting SLA tothe Delegation Daemon on the front-end node.Therefore, the requesting scheduler has to be ableto communicate with the daemon’s WSAG in-terface. The Delegation Daemon starts the VFEprior to the beginning of the negotiated time slice.The job submission and monitoring are then doneby the middleware interface of the virtual front-end. After the end of the negotiated time slice, theDelegation Daemon shuts down the VFE.
370 G. Birkenheuer et al.
Fig. 6 The sequenceof a resource delegationon the basis of theaforedescribedvirtualization approach.After negotiation, the DD(using the components asdescribed above) has todeploy the middlewareinfrastructure and startthe VMs. At the starttime of the negotiateddelegation, the LRMdaemons are started onthe underlying computenodes. The requestingscheduler can then accessthe resources through thenewly deployed services.At delegation end time,all services areautomatically shut down
Requesting Scheduler
virtual Frontend
ProvidingScheduler
DelegationDaemon
commit SLA
execute jobs
cluster-node
deploymw/lrms/node deamon
reserve negotiatedresources
start VFE
submit jobs
shut downVFE
results
run job
fetchresults
submit negotiation result
begin ofnegotiated time slice
end ofnegotiated time slice
fetchVM
commit SLA
start LRMSdaemons
5.4.2 Basic Setup of the Middleware Image
The resource provider has two options for pro-visioning the VFE: It can either be started as aVM on an existing front-end or on one of thedelegated compute nodes. The first method allowsdistributing real IP addresses, so the virtual front-end can be contacted from any Internet client and,therefore, from the scheduler of the requestingcommunity. The second option, which is morescalable, is to not use a dedicated server but one
of the delegated worker nodes as virtual front-end. On most sites this implies to have an internalIP in a network address translation (NAT) envi-ronment. In this case the virtual front-end has toset up a tunnel into the network of the requestingcommunity, and the tunnel has to use a real IP ad-dress out of the range of the requesting commu-nity’s network. IP administration and certificatehandling have to be handled by the requestingcommunity. Virtually, the front-end runs in theenvironment of the requesting community.
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 371
The existing front-end running the Grid mid-dleware is often not identical with the host thatruns the LRMS. For instance, in the D-Grid Ref-erence Installation [1], there is no connectivitybetween the front-end and the compute nodes.However, this is presumably necessary for theVFE depending on the setup of the LRMS usedwithin a delegation.
The VM image of the virtual front-end couldeither be a predefined image provided by theresource provider or a custom image providedby the requesting community. In the first case,the site provides a set of VMs running standardGrid middleware environments. The second caseprovides much more flexibility for the request-ing community but constitutes a security risk forthe providing site because the VFE has accessto the internal network. Additionally, this casealways implies that the requesting community getsadministrative privileges on the VFE. Therefore,the decision to support this scenario has to becarefully weighted by each site.
The resource provider or the requesting com-munity can provide the DNS name and the cor-responding host certificate of the VFE. The lattertakes place in case the requesting community alsoprovides the VM image of the VFE. The providingparty offers the IP address of the VFE and shouldreserve a set of IP addresses to support multipleresource delegations.
It is necessary that the certificates involved in aresource delegation process, e.g., of the requestingscheduler, the VFE, and users are issued by a CAthe resource provider trusts. For the first genera-tion implementation inside the D-Grid infrastruc-ture, it is also required that the requesting VO andits users already exist at the providing site. Thereare several options to handle the managementof the LRMS within a delegation scenario. Oneoption is to start a new LRMS server (e.g., a PBSserver) on the VFE, which helps to separate thedelegated resources from the non-delegated ones.The LRMS daemons (e.g., the mom in PBS-basedsystems) running on the worker nodes are eitherthe daemons running on the delegated nodes ornewly started job daemons. In the first case, theexisting daemons have to be disconnected fromtheir server, which not only is very intrusive fromthe siteÅRs perspective but might also be compli-
cated. In the second case, the new LRMS daemonscould run parallel to the existing daemons or theycould be started as jobs by the existing daemons.The latter is the preferred approach because usingthe existing daemons to start the new infrastruc-ture ensures that the new infrastructure is au-tomatically destroyed when the negotiated timeslice ends.
Another aspect is the data management on thedelegated resources. In accordance with the D-Grid Reference Installation, the VFE must pro-vide home directories for the users of the re-questing community, which must also exist on thecompute nodes. Additionally, scratch directoriesare necessary. For the first generation implemen-tation, the existing directories are used just ason the normal front-end. If these directories aremounted on the compute nodes, the job resultsare automatically available on the VFE. Further-more, the virtualization approach can also provideresources with additional application software. Tothis end, the VFE could export directories to thecompute nodes within a distributed file system.Alternatively, VMs running on the compute nodescould already contain necessary software. At theend of the negotiated time slice, the DelegationDaemon can wait a short period of time beforeit shuts down the VFE. The requesting schedulercan use this time for final data transfers. If acustom VFE was used, the requesting schedulermight also want to retrieve the VM image aftershutdown.
Eventually, a delegation setup is delivered asdepicted in Fig. 7.
5.5 Employed Technologies
As Section 5 discusses several alternative ap-proaches that are generally suitable to implementthe resource delegation process, the followingparagraph gives a brief overview of the technolo-gies chosen for the implementation in DGSI.
For the implementation and steering of the vir-tual machines, we use libvirt, which allows thetransparent management of several virtualizationtechnologies, like Xen, VMware ESX, KVM, andVirtualBox. Within the project, we focus on Xenand KVM. The WS-Agreement implementationis based on the WSAG4J Java framework. The
372 G. Birkenheuer et al.
Fig. 7 A delegationscenario based on avirtualized front-endnode. After negotiatingthe conditions of thedelegation (step 1),the providing schedulernotifies the DelegationDaemon to prepare thedelegation (step 2),which in turn setsup and starts the virtualfront-end. Then, therequesting schedulercan access the delegatedresources through themiddleware as requestedin the Service DescriptionTerms
ProvidingScheduler
RequestingScheduler
Resources for delegation
DD:Grid MW-Globus
-UNICORE-gLite
RMSPBSSGE
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
ComputeNode
1. Negotiationfor resourcedelegation
2. Preparedelegation
3. Setup and start
of theVFE
4. access to the delegated Grid
ressourcesvia VFE
Virtual Frontend
MiddlewareLRMS Server
connection of these through the VirtualizationManager is delivered by the libvirt Java APIbindings.
In DGSI, we focus on Globus Toolkit 4and UNICORE 6 as supported middleware sys-tems. On the LRMS level, we (although beingtechnology-agnostic) support PBS (TORQUE/Maui), /emphLSF, and SGE.
6 Information Model
The DGSI information model is designed to sup-port the three phases of a delegation: discov-ery of resources and meta-schedulers, negotiation,and monitoring. This includes a description of re-sources, meta-schedulers, constraints, monitoringdata and configuration state of resource delega-tions. For example, the constraints of a delegationrequest may require four machines with 8 CPUcores, 2GB memory/CPU core and 10GB tempo-rary disk space.
In DGSI, we use GLUE 2.0 [2] as a basis forresource descriptions. GLUE annotates resourceswith both static configuration (e.g., machine prop-
erties such as total amount of memory and num-ber of cores) as well as dynamically changingattributes including system queue length or cur-rent memory usage. The description of resourcesis necessary for three parts of the delegationprotocol:
1. Meta-scheduler discovery,2. matching the capabilities of a resource with
the requested resources, and3. static configuration.
First, resource descriptions are aggregatedto describe the capabilities of a single meta-scheduler and used for meta-scheduler discovery.A simple strategy for discovering an appropriatescheduler is to use a global resource registry for allmeta-schedulers. This approach, however, intro-duces a single-point of failure and is not scalable.We are currently investigating how resource dis-covery can be done with an eventually consistent,low traffic, gossiping protocol [23].
Second, as part of the WS-Agreement protocol(see Fig. 4), each providing scheduler maintainsa set of templates indicating its capabilities. InDGSI, these templates are in JSDL [4] extended
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 373
with GLUE to provide the user with a moreflexible resource description format. Finally, weuse GLUE to describe the configuration of thedelegated resource. This information is passed tothe requesting scheduler via the WS-Agreementprotocol as part of a template.
JSDL is used by the requesting scheduler tomodel the requirements of the resource. However,since JSDL only provides a limited number of at-tributes for describing resources, we extend JSDLwith a GLUE document.
To complete our information model, we needa way to describe time constraints of delegations.Since this is not part of JSDL or GLUE, we usethe Advance Reservation Profile (ARP) currentlydeveloped by the GRAAP-WG in the OGF.3 TheARP allows simple reservations of the type “3hours anywhere between 15:00 to 23:00”.
7 Performance Evaluation
The prototype for the resource delegation ap-proach described in this paper has been imple-mented in generation one of the DGSI virtu-alization approach. The prototype demonstratesthe general feasibility of resource delegations.The managing software for the resource delega-tion is based on a Java implementation of WS-Agreement (WSAG4J) [40]. The prototype canreserve the requested resources on a torque/mauiRMS and create and manage KVM based virtualfront-ends. Abilities of the control manager, likedynamic firewall configuration and identity man-agement, have not yet been implemented. Thenecessary firewall-ports have to be configured sta-tically, and the necessary certificates have to bedeployed manually. The file-staging daemon is notavailable, and the VM-Repository is limited. Thefirst prototype VFE supports Globus Toolkit 4.0.8images.
This section presents performance measuresand evaluates the abilities and throughput of a
3The Advance Reservation Profile Document is underactive development on the group’s collaboration space, butyet to be published as a full proposed recommendationdocument.
resource delegation. The test scenario had threeload tests.
1. load test with 100 times one RD negotiation2. load test with 100 times two parallel RD nego-
tiations3. load test with 100 times four parallel RD ne-
gotiations
Every negotiation was encapsulated in a javarun() method. The time for the negotiation wasmeasured taking the system time before anddirectly after the negotiation request. The nego-tiation was requested from the Paderborn Cen-ter for Parallel Computing. The goegrid clus-ter system in Göttingen created the reservation.The results of the negotiation tests are shown inFig. 8.
This border marks in the figure are noconfidence intervals. Through a lot of measure-ments, confidence intervals become very narrow,even when the measurements are scattered, be-cause for many measurements there is a highprobability that the average is beneath the cal-culated average. We want to show that the mea-surements are varying; the borders show the areawhere 95% of the measurements fall into. Theresults show that the duration of a negotiation
0
10
20
30
40
50
60
70
80
One request
Two requests
Four requests
Tim
e in
sec
onds
Fig. 8 The duration of the negotiations for a resourcedelegation
374 G. Birkenheuer et al.
increases linearly in the number of parallel exe-cutions. The average duration of the tests was:
1. battery: 18.609 ms2. battery: 34.221 ms3. battery: 67.235 ms
The long duration is caused by the time theVFE reservation and setup and the maui reserva-tion take. The VFE creation includes the cloningof a virtual machine that uses several seconds.The VFE creation and the reservations have prob-lems with parallel execution and are synchronized,explaining the increasing execution time duringthe parallel negotiations. To underline this, Fig. 9shows server internal measurements of the nego-tiation process.
– Strategy duration: mean of 20.271 ms– RMS reservation: mean of 44.556 ms– VFE reservation: mean of 46.848 ms– Handler duration: mean of 30.949 ms– VFE setup: mean of 12.653 ms
The server side negotiation process uses twomethods. Firstly the strategy method, which re-serves the resources on the RMS and a time slotfor the VFE. Secondly, the handler method iscalled and starts the VFE setup when the strat-egy method successfully reserved the resources.Thus, in Fig. 9 the Strategy duration includes theduration time for creating the RMS and VFE
0
5
10
15
20
25
30
35
40
45
50
Strategy duration
RMS reservation
VFE reservation
Handler durationVFE setup
Tim
e in
sec
onds
Fig. 9 The duration of the server side components
reservation. The Handler duration includes thetime for running the VFE setup. Again, the figuresindicating the area 95% of the measurements fellinto. One can see that the VFE setup of a RDcreation takes 12 seconds on average. The handlerduration takes between from 12 to 50 s. This isdue to the fact that the call of the VFE setup issynchronized and leads to a huge delay of thehandler duration when parallel RD negotiationsare called. The effect for the Strategy durationis similar. The huge percentiles show that singlenegotiations are performant and that, due to se-rialization, the duration increases by concurrentnegotiations’. This underlines why the single RDnegations are much faster than concurrent negoti-ations executions.
However, the first prototype demonstrates thatRD negotiations are possible. For the next gen-eration of the software, we plan to migrate thenegotiation process to the software stack, of theSLA4DGrid project, which includes a sophisti-cated WS-Agreement stack, and to submit theVFE setup as a separate job. This process shouldimprove the responsiveness of the negotiationprocess because the duration of the negotiation inthe generation two RD software stack will consistof the strategy method only.
8 Conclusion and Future Work
Driven by user communities and resource pro-viders, DGSI implements two use-case models:Delegation of activities and delegation of re-sources; both were not available before in D-Grid. Whereas delegation of activities means job-forwarding at this point in time, delegation ofresources implements a Cloud usage style forGrid resources. The control on a set of resourcesis handed over to the user for the negotiatedtime. The genuine innovation of DGSI is thedefinition of the protocol suite to negotiate del-egations on activities (jobs) and resources. Todescribe these protocols, DGSI harvests on Gridstandards and delivers a confirmation of valueof systematic OGF work over many years. TheDGSI functionalities are achieved by reuse ofproven technologies in a practical, straightforwardway. Basically, DGSI recombines a number of
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 375
reference implementations and meta-schedulers.Besides implementing missing parts of the archi-tecture and providing adequate security mecha-nisms for resource delegation, at the heart of thepractical technical work for the creation of creat-ing some GLUE(ing) elements, JSDL profiles andWS-Agreement templates, and the performanceof scrutinized QA testing to achieve productionreadiness. In order to show real-world applica-bility, we have evaluated the described approachin a real-world testbed, and recorded promisingperformance results.
DGSI is Path-breaking for Grid and Cloud FutureOriginally, DGSI was set up as a gap project for D-Grid, closing identified functionality gaps withinthe multi-technology environment of the Ger-man e-science Grid. DGSI achieves this goal bystrict (re-)use of standards, enabling all technol-ogy providers (be it Grid or Cloud middleware) toadopt the methods and protocols and implementnegotiated delegations. By experience, the Gridcommunity can state: There will never be a bigenough Grid. The same applies to Clouds, whereother criteria add to the picture. DGSI realizesthe integration of Grid and Cloud technology byproviding the technology to dynamically extenda local Grid with remote resources hosted in adifferent administrative domain. For both, Gridsand Clouds, nobody seriously envisions a futurewhere most or even all run the same softwarestack, using the same middleware. Technologicaldiversity will dominate—which is good: systemsbiology taught us about the stabilizing effectsof diversity on ecosystems. DGSI delivers theblue-print to utilize technological diversity. Thestandards-based methods on negotiation and del-egation do not limit the creativity of the Cloudor Grid middleware developers while providinginteroperability with other technology stacks (e.g.legacy).
DGSI’s current agenda reaches to the pointwhere several meta-schedulers and LRMS aredelegation-enabled and those methods are intro-duced into D-Grid operations. Through contin-uous active contributions to the OGF workinggroups, DGSI takes care to keep innovative el-ements in line with standardization. Beyond orbeside the current project and as part of future
work, we are looking for partners to work onCloud stacks and potentially more Grid technolo-gies. DGSI blue-prints may be the starting pointand base line for interoperability in diversity. TheDGSI partners are offering help to implement therelated protocols and standards and to enhancedelegation methods to suit innovative use cases.
We envision a world of distributed computingwith resources from many different providers us-ing various technologies, where borders betweenClouds and Grids are blurred or vanishing, wherenew technologies are introduced without serviceinterruption, where innovation is free to flourishwhile staying productively connected, fully inter-operable.
References
1. Alef, M., Fieseler, T., Freitag, S., Garcia, A.,Grimm, C., Gürich, W., Mehammed, H., Schley, L.,Schneider, O., Volpato, G.: Integration of multiplemiddlewares on a single computing resource. FutureGener. Comput. Syst. 25(3), 268–274 (2009). doi:10.1016/j.future.2008.05.004
2. Andreozzi, S., Burke, S., Ehm, F., Field, L., Galang,G., Konya, B., Litmaath, M., Millar, P., Navarro,J.: GLUE Specification v. 2.0 (2009). Grid Fo-rum proposed recommendation GFD.147, available athttp://www.ogf.org/documents/GFD.147.pdf
3. Andrieux, A., Czajkowski, K., Dan, A., Keahey, K.,Ludwig, H., Nakata, T., Pruyne, J., Rofrano, J.,Tuecke, S., Xu, M.: Web Services AgreementSpecification (WS-Agreement). Grid Forum proposedrecommendation GFD.107, available at http://www.ogf.org/documents/GFD.107.pdf
4. Anjomshoaa, A., Brisard, F., Drescher, M., Fellows,D., Ly, A., McGough, S., Pulsipher, D., Savva,A.: Job Submission Description Language (JSDL)Specification v1.0 (2008). Grid Forum recommen-dation GFD.136, available at http://www.ogf.org/documents/GFD.136.pdf
5. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D.,Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A.,Rabkin, A., Stoica, I., Zaharia, M.: Above the Clouds:A Berkeley View of Cloud Computing. Tech. rep.,EECS Department, University of California, Berkeley(2009)
6. Battré, D., Hovestadt, M., Wäldrich, O.: Lessonslearned from implementing WS-Agreement. In:Wieder, P., Yahyapour, R., Ziegler, W. (eds.) Gridsand Service-Oriented Architectures for ServiceLevel Agreements, pp. 23–34. Springer US (2010).doi:10.1007/978-1-4419-7320-7_3
7. Baur, T., Breu, R., Kálmán, T., Lindinger, T.,Milbert, A., Poghosyan, G., Reiser, H., Romberg,M.: An interoperable grid information system for
376 G. Birkenheuer et al.
integrated resource monitoring based on virtual or-ganizations. J. Grid Computing 7, 319–333 (2009).doi:10.1007/s10723-009-9134-3
8. Bazaar: Project Website: https://www.ce-egee.org/weblog/bazaar (2010)
9. Bolte, M., Sievers, M., Birkenheuer, G.,Niehoerster, O., Brinkmann, A.: Non-intrusivevirtualization management using libvirt. In:Proceedings of Design, Automation and Test inEurope (DATE). Dresden, Germany (2010)
10. BREIN—Business Objective Driven Reliable and In-telligent Grids for Real Business: Project Website:http://www.eu-brein.com (2010)
11. Cornwall, L.A., Jensen, J., Kelsey, D.P., Frohner, Á.,Kouril, D., Bonnassieux, F., Nicoud, S., Lorentey,K., Hahkala, J., Silander, M., Cecchini, R.,Ciaschini, V., dell’Agnello, L., Spataro, F.,O’Callaghan, D., Mulmo, O., Volpato, G.L., Groep,D., Steenbakkers, M., McNab, A.: Authenticationand authorization mechanisms for multi-domain gridenvironments. J. Grid Computing 2, 301–311 (2004).doi:10.1007/s10723-004-8182-y
12. England, D., Weissman, J.B.: Costs and benefits of loadsharing in the computational Grid. In: Feitelson, D.G.,Rudolph, L., Schwiegelshohn, U. (eds.) Proceedings ofthe 10th Job Scheduling Strategies for Parallel Process-ing, Lecture Notes in Computer Science (LNCS), vol.3277, pp. 160–175. Springer (2004)
13. Ernemann, C., Hamscher, V., Schwiegelshohn, U.,Streit, A., Yahyapour, R.: On advantages of Grid com-puting for parallel job scheduling. In: Proceedings ofthe 2nd IEEE/ACM International Symposium on Clus-ter Computing and the Grid (CCGRID02), pp. 39–46.IEEE Press (2002)
14. Eucalyptus Systems: Eucalyptus. Website: http://open.eucalyptus.com/ (2010)
15. Field, L., Laure, E., Schulz, M.: Grid deployment ex-periences: Grid interoperation. J. Grid Computing 7,287–296 (2009). doi:10.1007/s10723-009-9128-1
16. Fölling, A., Grimme, C., Lepping, J., Papaspyrou, A.:Decentralized Grid scheduling with evolutionary fuzzysystems. In: Frachtenberg, E., Schwiegelshohn, U.(eds.) Proceedings of the 14th Job Scheduling Strate-gies for Parallel Processing, Lecture Notes in Com-puter Science (LNCS), vol. 5798, pp. 16–36. Springer(2009)
17. Fölling, A., Grimme, C., Lepping, J., Papaspyrou,A., Schwiegelshohn, U.: Competitive co-evolutionarylearning of fuzzy systems for job exchange in computa-tional Grids. Evol. Comput. 17(4), 545–560 (2009)
18. Gagliardi, F., Jones, B., Grey, F., Begin, M.E.,Heikkurinen, M.: Building an infrastructure for scien-tific Grid computing: status and goals of the EGEEproject. Phil. Trans., Ser. A, Math. Phys. Eng. Sci.363(1833), 1729–1742 (2005)
19. GRAAP-WG—Grid Resource Allocation AgreementProtocol Working Group): Project Website: https://forge.gridforum.org/projects/graap-wg (2010)
20. Grimme, C., Lepping, J., Papaspyrou, A.: Prospects ofcollaboration between compute providers by means ofjob interchange. In: Frachtenberg, E., Schwiegelshohn,
U. (eds.) Proceedings of Job Scheduling Strategies forParallel Processing, Lecture Notes in Computer Sci-ence (LNCS), vol. 4942, pp. 132–151. Springer (2007)
21. GSA-RG—Grid Scheduling Architecture ResearchGroup: Project Website: https://forge.gridforum.org/projects/gsa-rg (2009)
22. Iosup, A., Epema, D.H., Tannenbaum, T., Farrellee,M., Livny, M.: Inter-operating Grids through delegatedmatchmaking. In: Proceedings of International Confer-ence for High Performance Computing, Networking,Storage and Analysis (SC07). Reno, NV (2007)
23. Iyengar, V., Tilak, S., Lewis, M.J., Abu-Ghazaleh, N.B.:Non-uniform information dissemination for dynamicGrid resource discovery. In: NCA ’04: Proceedings ofthe Network Computing and Applications, Third IEEEInternational Symposium, pp. 97–106. IEEE ComputerSociety, Washington, DC (2004)
24. Kota, S.R., Shekhar, C., Kokkula, A., Toshniwal, D.,Kartikeyan, M.V., Joshi, R.C.: Parameterized modulescheduling algorithm for reconfigurable computing sys-tems. In: ADCOM ’07: Proceedings of the 15th In-ternational Conference on Advanced Computing andCommunications, pp. 473–478. IEEE Computer Soci-ety, Washington, DC (2007)
25. Kurowski, K., Nabrzyski, J., Oleksiak, A., Weglarz, J.:Scheduling jobs on the Grid—multicriteria approach.Comput Methods Sci Technol 12(2), 123–138 (2006)
26. Libvirt Development Team: Libvirt. Project Website:http://libvirt.org/ (2010)
27. NextGRID SLA Schema Generalized Specification:http://www.nextgrid.org/GS/management_systems/SLA_management/NextGRID_SLA_schema.pdf (2009)
28. Nimbus Community: Nimbus. Project Website: http://www.nimbusproject.org/ (2010)
29. OpenNebula Community: OpenNebula:. ProjectWebsite: http://www.opennebula.org/ (2010)
30. Parkin, M., Badia, R.M., Martrat, J.: A Comparisonof SLA Use in Six of the European Commissions FP6Projects. Tech. Rep. TR-0129, Institute on ResourceManagement and Scheduling, CoreGRID—Networkof Excellence. URL http://www.coregrid.net/mambo/images/stories/TechnicalReports/tr-0129.pdf (2008)
31. PHOSPHORUS—Lambda User Controlled Infra-structure For European Research:. Project Website:www.ist-phosphorus.eu (2010)
32. Riedel, M., Terstyanszky, G.: Grid interoperabilityfor e-research. J. Grid Computing 7, 285–286 (2009).doi:10.1007/s10723-009-9138-z
33. Rings, T., Caryer, G., Gallop, J., Grabowski, J.,Kovacikova, T., Schulz, S., Stokes-Rees, I.: Grid andcloud computing: opportunities for integration with thenext generation network. J. Grid Computing 7, 375–393(2009). doi:10.1007/s10723-009-9132-5
34. Rumpl, A., Wäldrich, O., Ziegler, W.: Extending WS-Agreement with multi-round negotiation capability. In:Grids and Service-Oriented Architectures for ServiceLevel Agreements, CoreGRID series 13, pp. 89–103.Springer (2010). doi:10.1007/978-1-4419-7320-7_9
35. Seidel, J., Wäldrich, O., Wieder, P., Yahyapour, R.,Ziegler, W.: SLA for resource management andscheduling—a survey. In: Grid Middleware and Ser-
Infrastructure Federation Through VirtualizedDelegation of Resources and Services 377
vices: Challenges and Solutions, CoreGRID Series 8.Springer (2008)
36. SLA@SOI—Project Website:. Project Website: http://sla-at-soi.eu (2010)
37. SmartLM—Grid-friendly Software Licensing for Lo-cation Independent Application Execution:. ProjectWebsite: http://www.smartlm.eu (2010)
38. SORMA—Self-Organizing ICT Resource Manage-ment:. Project Website: http://www.im.uni-karlsruhe.de/sorma (2010)
39. Subramaniyan, R., Troxel, I., George, A.D., Smith, M.:Simulative analysis of dynamic scheduling heuristics
for reconfigurable computing of parallel applications.In: Proceedings of the 2006 ACM/SIGDA 14th Inter-national Symposium on Field Programmable Gate Ar-rays, pp. 230–230. ACM (2006)
40. Wäldrich, O.: WS-Agreement for JAVA (WSAG4J):Project Website: http://packcs-e0.scai.fraunhofer.de/wsag4j/
41. IBM Web Service Level Agreements (WSLA) Project:Project Website: http://www.research.ibm.com/wsla
42. OASIS Web Services Resource Framework (WSRF)TC: Project Website: http://www.oasis-open.org/committees/wsrf
