1 CENTER FOR PARALLEL COMPUTERS DEPARTMENT OF COMPUTER SCIENCE DEPARTMENT OF COMPUTING SCIENCE 2 nd International Conference on Service Oriented Computing,

1

CENTER FORPARALLEL

COMPUTERS

DEPARTMENT OF COMPUTER SCIENCE

DEPARTMENT OFCOMPUTING SCIENCE

2nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004

An OGSA-Based Accounting System for Allocation Enforcement across HPC Centers

TS10 – Service Applications

Thomas Sandholm [email protected] MulmoPeter GardfjällErik ElmrothLennart Johnsson


Key Question

?How do we share national Grid compute resources in a fair, secure, open, and

scalable way


Outline

• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A


Fairness vs. Utilization

• Trade-off: Fair resource distribution and optimal resource utilization

• Soft real-time quota enforcement• User preferences• Resource policies• Allocation authority policies

Fair Distribution

Maximum Utilization

SecureOperation

Scalable Efficiency


Security vs. Scalability

• Integrity & Privacy• Single Sign-on/Impersonation• DoS/Replay Attack prevention• Privilege Delegation• Message Level vs. Transport Level• Policy Driven Authorization: PDP, PAP, PIP, PEPScale:• National Grid • No single point of failure but

coordinated allocationenforcement

Fair Distribution

Maximum Utilization

SecureOperation

Scalable Efficiency


Openness & Interoperability

• Systems Integration Platform• Scheduler/Workload Manager Agnostic• Programming Language/Model Agnostic• Portable (100% pure Java)• XML Based Standards: XPath, XQuery, XSLT, GGF-

UR, XML-Signature, XML-Encryption, XACML• Web/Grid Services Standards: SOAP, WSDL, WS-

Security, OGSA, GGF-UR, GSI, GSSAPI, OGSI/WSRF


Outline



Open Grid Services Architecture• Global Grid Forum Standardization Initiative• Architecture extending SOA and WSA to dynamically

share stateful resources across organizational boundaries (=realizing the Grid vision)

• “… defining, within a service-oriented architecture, a set of core capabilities and behaviors that address key concerns in Grid systems.” OGSAv1

• Assumes state modeling according to OGSI/WSRF Core Infrastructure offering Inspection,

Discovery, Lifetime Management, Notifications, Fault Handling

• WS-Resource = stateful resource and associated Web service. Provide context for message exchange

• Addresses Grid security requirements such as Delegation and Single Sign-On


Role of Accounting in OGSA

Accounting foundational service to: • Job Execution

Make sure that only jobs with sufficient quota can be executed on the compute resource

Decide queue priority based on available funds and usage history

• QoS/SLA Management Negotiate pricing based on resource usage Optimizing Utilization SLA Attainment/Policing

• Security Auditing Access Control PEP/PIP


Outline

• Requirements on Software Qualities • Open Grid Services Architecture • SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A


SweGrid

• SweGrid connects 600 compute nodes (Intel P4) across 6 Swedish HPC centers interconnected by 10Gbs GigaSunet network

• 400 HPC users at all centers (some overlapping)

• Inaugurated March 2004• ~50 currently active researchers • Up to 10k jobs per month per site


SweGrid Continued

• Resource quotas allocated by Swedish National Allocations Committee (SNAC) after peer-review of promising research projects with high computational demands (c.f. NRAC)

• Initially homogeneous hardware but heterogeneous scheduling, security, and accounting environment (policies, tools, data, processes, etc)

• Wanted: Uniform resource quota use & allocation


Outline



SweGrid Accounting System (SGAS) Key Design Points

1. Decentralized accounting solution based on standard, open protocols in compliance with the proposed OGSA

2. 3-party (user, resource, allocation authority) policy customization

3. Non-intrusive to local site accounting systems4. All components governed by a scalable cross-

organizational authorization framework


SGAS Component Overview

PAT

Bank LUTS

Resource

Scheduler

Resource Manager

BrokerUser

WSDL WSDL

WSDL WSDL

JARM

Policy Administration ToolLogging and Usage Tracking ServiceJob Account Reservation Manager

SubmitJob

Reserve/Release PublishUR

QueryAddUser


SGAS Security Design

PAT

Bank

Resource

Scheduler

Resource Manager

BrokerUser

JARMPEP

PDP

PAP

Site Policy Manager

PIP

PDP

LUTSPIP

PAP

External Authorization Service

PDP

Membership/CommunityService

PIP

Policy Administration PointPolicy Decision PointPolicy Information PointPolicy Enforcement Point

Credential Delegation

WS-SecureConversation

XML-SignatureXML-Encryption

PKI

Kerberos


Outline



Overdraft XACML Policy

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97

175%

150%

125%0

20000

40000

60000

80000

100000

120000

140000

Execution Time (ms)

Jobs

Overdraft Limit

<Condition FunctionId= "urn:oasis:names:tc:xacml:1.0:function:integer-less-than-or-equal"> <Apply FunctionId= "urn:oasis:names:tc:xacml:1.0:function:integer-one-and-only"> <EnvironmentAttributeDesignator AttributeId= "sgas:overdraw:percent:requested" DataType= "http://www.w3.org/2001/XMLSchema#integer"/> </Apply> <AttributeValue DataType= "http://www.w3.org/2001/XMLSchema#integer"> 175 </AttributeValue></Condition>


Overdraft Fuzzy Logic Policy

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97

Fair

Fuzzy

Unfair0

20000

40000

60000

80000

100000

120000

140000

Execution Time (ms)

Jobs

Submission Flow

R1: overdraft is low allocation left is much allow reservationR2: overdraft is high allocation left is little disallow reservationR3: allocation proximity is soon overdraft is high allocation left is much allow reservationR4: allocation proximity is soon overdraft is low allocation left is little allow reservation


Super Computing 2004 Demonstration


Outline



Conclusions

• Document centric communication in conjunction with semi-structured native XML databases is a very flexible combination

• Batch charging and eager prepare reservation needed for scalability

• Timestamp based allocations distributed in a staggered monthly flow result in the best trade-off between fairness and utilization

• Generic PEP/PDP/PIP/PAP model useful for encapsulating and evolving authorization code

• OGSI/WSRF state management ideal for controlling fine grained service state such as account quotas, reservations and policies in a standard way


Future Work

• With large-scale flexibility and configurability comes complexity and it becomes hard to optimize high-level goals and to realize detailed user QoS requirements – development of an SLA Management framework and user/resource goal driven optimizing agents (WS-Agreement, ContractNet)

• Initial focus has been on scientific community resource sharing - support economic brokering and for-profit banks

• Multi jobs may overload the bank - SAML assertions (c.f. cheques) as a multi-allocation payment and reservation method


Learn more…

http://www.sgas.se

http://www.swegrid.se


Outline


Documents

1 CENTER FOR PARALLEL COMPUTERS DEPARTMENT OF COMPUTER SCIENCE DEPARTMENT OF COMPUTING SCIENCE 2 nd International Conference on Service Oriented Computing,