Upload
sandra-reynolds
View
221
Download
3
Tags:
Embed Size (px)
Citation preview
Grid Tools
Jai Dayal & Casey Rathbone
Outline Web Services Unicore BOINC Globus Condor
Web Services“A Web service is an abstract notion that must be implemented by a concrete agent. … The agent is the concrete piece of software or hardware that sends and receives messages, while the service is the resource characterized by the abstract set of functionality that is provided.”
Web Services
Web Services
Remote Procedure Calls (RPC) - distributed function call interface
Service Oriented Architecture (SOA) - utilizes message passing to communicate to services also known as "message-oriented" services
Representational State Transfer (REST) - emulates HTTP (or similar protocols) to communicate stateful resources
Unicore - Overview Single point of entry – security
Uses SSL
Uses web services SOAP Messages WSDL
Reputation for simple deployment
Unicore - Architecture
Most basic resourceA cluster, a PC, web service, etc
Vsite
Unicore - Architecture The gateway is the point of entry to the
Unicore grid
Vsites
Gateway
Unicore - Architecture Gateway + Vsites = Usite
Vsites
Gateway
Usite
Unicore - ArchitectureUNICORE - Architecture
Vsites
Gateway
Usite
Client
Unicore - Architecture
Vsites
Gateway
Usite
Client
Client communicates using SOAP SOAP/HTTP binding
SOAP Message
SOAP contains: Address of
gateway Port number Vsite name task descriptions
Unicore - Security• Gateway handles authentication
– SSL used to authenticate the client– SSL also used by client to authenticate
gateway– Both present digital certificates
• Vsite handles authorization– Gateway passes along certificate in SOAP header– If client has privileges, Vsite authorizes
•This model allows end to end security
• Removes some strain from gateway
• Trust between gateway and Vsite
BOINC - Overview
Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust
BOINC - Overview
Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust
BOINC GoalsCheap and easyProject varietyPublic participation → reward points
BOINC - Overview Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust BOINC Goals
Cheap and easyProject varietyPublic participation → reward points
BOINC projectsSETI at homeFolding at homePredictor at home
BOINC - ArchitectureBOINC
BOINC - ArchitectureBOINCProjects
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
Tools
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
Tools
Client
BOINC - Workflow Client contacts Scheduling Server
Server returns XML Document
Workunit: inputs for a computation/jobApplicationsInput filesEnvironment variables
Client runs job and places results in output files
Result:Reference to workunitReference to output files
BOINC – Fault tolerance
TransitionerIdentifies bad results
ValidatorDetermines canonical result
Public Resource Computing is volatileUsers leave before job completingOver-clocking can lead to bad resultsMalicious users
AssimilatorHandles canonical result
These daemons can run on different hosts
Homogenous RedundancyComputer architecture specific jobs
BOINC – Client settings Client implements “local scheduling policy”
Utilizes maximum resourcesMeet job deadlinesRespect clients resource allocation wishesOffers some variety to the client
Anonymous platformUsed for open source projectsClients can modify apps to fit their resourcesCommunicate with scheduling server using XML config file.
Globus Toolkit 4
• GT4 is a open-source toolkit developed to build grids • It provides full capabilities for sharing computing power
and databases • Usage of Globus is extensive throughout the scientific
community within NSF, DOE, DARPA, IBM, NASA, and Microsoft projects
• The Large Hadron Collider project utilizes a Globus based grid for the European Data Grid.
Globus Toolkit 4
Globus Toolkit is a suite of Web-Services and Unix that controls provides an interface to submit, monitor, and cancel jobs on the Grid resources.
Globus Toolkit is an interface that allows for jobs to be submitted and executed at the appropriate resources within the grid
Globus Toolkit 4
Globus Toolkit 4 Security• GT4 uses (Grid Security Infrastructure) GSI with an public
key cryptography • GSI Functional Layers - Authorization, Authentications,
Delegation, Message Protection, and Message Format
Globus Toolkit 4 Data Management
• Data Management tools within the toolkit fall into either of two categories data replication and data movement
• Data Replication consists of Replica Location Service (RLS) • Data Movement consists of GridFTP and Reliable File
Transfer (RFT)
GT4: Data Replication
Replica Location Service provides the capability to track and maintain multiple locations of data across the grid. It is a distributed registry system that allows users and applications to register the locations of data.
GT4: Data Movement
• GridFTP - GT4 provides a server implementation, scriptable command-line interface and several GUI client interfaces
• Reliable File Transfer (RFT) - A Web Services Resource
Framework that schedules file transfers based on a set of criteria of when specific resources and bandwidth is available
Globus Toolkit 4 Execution Management
• Community Scheduler Framework - Grid level meta-scheduling framework
• Grid Teleoperation Control Protocol - A service interface that has been used to
• Dynamic Accounts - This service provides functionality to create groups and accounts for specific jobs that need to be submitted
• Grid Resource Allocation & Management - Used to locate, submit, monitor and cancel jobs on Grid computing resources. It is the fabric that ties all the schedulers and job queues together to utilize the resources.
GT4: Community Scheduler Framework
Open-Source implementation of OSGA-based meta scheduler
Provides an interface to existing grid platforms making integration more efficient
GT4: Community Scheduler Framework
The diagram below shows the basic layout of the scheduler framework
http://www.globus.org/grid_software/computation/csf.php
GT4: Grid Teleoperation Control Protocol
The GTCP is used to control heterogenous physical and computational simulations.It exposes the WSRF-compliant service interface used to control remote instruments and simulations and a interface to the backend GTCP to allow for additional control abstraction.
Globus Toolkit 4 Information Services
Monitoring and Discovery Service is a suite of web services used to discover resources and services on a grid
This service is dedicated to handling the complex task of organizing resources into Virtual Organizations (VO)
http://www.globus.org/toolkit/docs/4.0/info/key-index.html
Globus Toolkit 4 Common Runtime
GT4 contains a multitude of implementations of libraries allowing the suite to be platform independent.
• C Common Libraries• C WebService Libraries• Java WebService Libraries• Python WebService Libraries• XIO
Condor - Overview• University of Wisconsin – Madison• Open source• Linux, Solaris, Mac OS, Windows• No program modification
• Jobs tend to be architecture specific– Designed on x86 → job run on x86– Java applications possible exception
• Overall concept is simple– Install it on each desired machine– Designate one as central manager
• Practice is much harder– Installation is tricky– Lots of commands to learn
Condor• Central Manager
– Detects idle machines– Matches job with available resources– Considers each processor as separate resource
• Jobs– Submitted via “submit file”– Universe, executable name, inputs, outputs, error, log file
• Sequential Jobs– Standard universe
• Used for long jobs (weeks, months)• Allows partial execution, pausing• Can relocate job to another machine
– Vanilla universe• For short term jobs
– Local universe• Submitter executes job
• Parallel Jobs– Supports MPI and PVM
Condor - variety• Flocking
– Connects multiple pools– If current pool has no resources, job “flocks”– Set “flock_to” variable in config file
• Condor-G– Distributed flocking– Utilizes Globus– Universe = Globus– Also specify “globusscheduler”
• Points to the globus job manager
References[1] D. P. Anderson, BOINC: A System for Public-Resource Computing and Storage. In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. November, 2004.
[2] S. Bagchi, E. Hung, A. Iyengar, N. Vogl, and N. Wadia, Capacity Planning Tools for Web and Grid Environments. In Proceedings of the 1st international conference on Performance evaluation methodolgies and tools. October, 2006.
[3] F. Hernández, P. Bangalore, K. Reilly, End-User Tools for Grid Computing. In Proceedings of the first workshop on End-user software engineering. July, 2005.
[4]I. Habib, Getting Started with Condor. Linux Journal, Volume 2006, Issue 149, September 2006.
[5]Menday, R., "The Web Services Architecture and the UNICORE Gateway," Telecommunications, 2006. AICT-ICIW '06. International Conference on Internet and Web Applications and Services/Advanced International Conference on , vol., no., pp. 134-134, 19-25 Feb. 2006
[6]Hellinckx, P.; Stuer, G.; Hendrickx, W.; Arickx, F.; Broeckhove, J., "Grid-user driven grid research, the CoBRA grid," Cluster Computing and the Grid Workshops, 2006. Sixth IEEE International Symposium on , vol.2, no., pp. 8 pp.-, 16-19 May 2006