View
7
Download
0
Category
Preview:
Citation preview
10/9/2013
1
CS 655 – Advanced Topics in Distributed Systems
Computer Science Department
Colorado State University
Presented by : Walid Budgaga
1
Outline
Condor
The Anatomy of the Grid
Globus Toolkit
2
Motivation High Throughput Computing (HTC)?
Large amounts of computing capacity over long
periods of time.
Measured: operations per month or per year
High Performance Computing (HPC)?
Large amounts of computing capacity for short
periods of time
Measured: FLOPS
3
Motivation
HTC is suitable for scientific research
Example(Parameter sweep):
Testing parameter combinations to
keep temp. at particular level
op(x,y,z) takes 10 hours, 500(MB) memory, I/O 100(MB)
x(100), y(50), z(25)=> 100x50x25=125,000(145 years)
4
Motivation Fort Collins
Science Center
Uses Condor for
scientific projects
Source: http://www.fort.usgs.gov/Condor/ComputingTimes.asp
5
HTC Environment
Large amounts of processing capacity? Exploiting computers on the network
Utilizing heterogeneous resources Overcoming differences of the platforms
By building portable solution
Including resource management framework
Over long periods of time? System must be reliable and maintainable
Surviving failures (software & hardware)
Allowing leaving and joining of resources at any time
Upgrading and configuring without significant downtimes
6
10/9/2013
2
HTC Environment
Also, the system must meet the needs of:
Resource owners
Rights respected
Policies enforced
Customers Benefit of additional processing capacity outweigh complexity of
usage
System administrators Real benefit provided to users outweigh the maintenance cost
7
HTC
Other considerations:
The distributive owned resources lead to:
Decentralized maintenance and configuration of resources
Resource availability
Applications preempted at any time
Adds an additional degree of resource heterogeneity
8
9
Condor Overview
Open-source high-throughput computing framework for computing intensive tasks.
Manages distributive owned resources to provide large amount of capacity
Developed at the Computer Sciences Department at the University of Wisconsin-Madison
Name changed to HTCondor in October 2012
10
Condor Overview
11
Condor Overview
12
Customer agent
Represent the customer job(application)
Can state the its requirement as following:
Need a Linux/x86 platform
Want the machine with the high memory capacity
Prefer a machine in the lab 120
10/9/2013
3
Condor Overview
13
Resource agent Represent the resource
Can state its offers as following: Platform: a Linux/x86 platform
Memory: 1GB
Can state its requirements as following:
Run jobs only when keyboard and mouse are idle for 15 m
Run jobs only from the computer department
Never run jobs belong to abc@cs.colostate.edu
Condor Overview
14
Matchmaker
Matches jobs and resources
based on requirements and offers
Notifies the agents when a match found
Challenges of HTC system:
Software Development
System Administration
15 16
Software Development Four primary challenges
Utilization of heterogeneous resources
Requires system portability.
Network Protocol Flexibility
Required to cope with constantly changing of the resource and customer needs
Required for adding new features
Remote file access
Required for giving ability for accessing data from any workstation
Utilization of non dedicated resources
Required for preempt and resume application.
17
Software Development
Utilization of heterogeneous resources:
Requires system portability obtained through layered system design
• Network API :
• Connection-oriented and connectionless
• Reliable and unreliable interfaces.
• Authentication and encryption
• Process management API :
• Create , suspend, resume,
and kill a process.
• Workstation statistics API:
• Reports information needed to
Implement resource owner policies
Verify the validation of the applications requirements
18
10/9/2013
4
Software Development
Network Protocol Flexibility:
To cope with adding new services in HTC without frequently updating HTC
components, general purpose data format may be used
• For example: Condor uses protocol similar to RPC
• Condor:
19
Software Development
Remote file access(1):
To guarantee that HTC applications can access their data
from any workstation in the cluster.
• Three possible solutions:
• Using existing distributed file system (NFS)
• Authenticates customer application,
• Privileges need to assigned, or
• Grant file access permission
20
Software Development
Remote file access (2):
• Implementing data file staging
• Transferring input and output files to remote workstation
specified by customer
• Require free disk space on workstation
• High cost for large data files
21
Software Development
Remote file access(3):
• Redirecting file I/O system calls
• Interposing HTC between application & operating system
• By Linking application with an interposition library
• Does not require file storage on remote workstation
• Reduce performance.
• Difficult to develop & maintain portable interposition
22
23
Software Development
Utilization of non-dedicated resources
Requires the ability for preempting and resuming application.
It can be obtained using checkpoints
Checkpoint:
snapshot of the state of the executing program
It can be used to restart the program at a later time
Provide reliability
Enable preemptive-resume scheduling
24
Software Development
Checkpoints in Condor (1)
Used as migration mechanism
Job scheduler to migrate jobs from workstations to others
Used to resume a vacated jobs
The program has the ability to checkpoint itself
Using a checkpointing library
To provide additional reliability
HTCondor can be configured to write checkpoints periodically
10/9/2013
5
25
Software Development
Checkpoints in Condor (2)
When checkpoints are stored:
Periodically, if HTCondor is configured
At any time by the program
When higher priority job has to start on the same machine
When the machine becomes busy
26
Software Development
Checkpoints in Condor (3)
Storing of checkpoints
By default,
checkpoints are stored on local disk of the machine
where job was submitted
However,
It can be configured to stored them on checkpoints server
27 28
System Administration
Administrator has to answer to:
Resource owners
By guaranteeing that HTC enforces their policies
Customers
By ensuring receipt of valuable services from HTC
Policy makers
By demonstrating that HTC is meeting the stated goals.
29
System Administration
Access Policies
Specifies when and how the resources can be accessed and
by whom
The policies might be specified using a set of expressions
For example in Condor:
Requirements (true: to start accessing the resources)
Rank (preference)
Suspend
Continue
Vacate (notification to stop using resources)
Kill (immediately stopping using the resources)
30
System Administration Access Policies
Example from Condor:
10/9/2013
6
31
System Administration
Reliability
The HTC must be prepared against failures and
must be automate failure recover for common failure.
It is not easy job
Detect difference between normal and abnormal termination
Don’t leave running applications unattended
Choose the correct checkpoint to restart
Decide when it is safe to restart the application
Determine & avoid bad nodes 32
System Administration
System logs
It is primary tools for diagnosing system failures. It gives the ability to reconstruct the events leading up to the failure .
Problems and Suggested solutions
Logs files can grow to unbounded size.
Detailed logs for recently events and summaries for old information
Managing distributed log file
Store logs centrally on a file server or a customized log server
Provide single interface by installing logging agents on each workstation
33
System Administration
Monitoring and Accounting
It helps the Administrator to:
Assess the current and historical state of the system
Track the system usage
CondorView Usage Graph
34
System Administration
35
System Administration
Security (1)
Possible attacks
Resource attack
Unauthorized user gains access to a resource
Authorized user violates resource owner’s access policy
Customer attack
Customer’s account or files are risked via HTC environment
36
System Administration
Security (2)
To protect against an unauthorized of resource access policy
Resource owner may specify authorized users in his access policy
Condor Example:
Requirement = (Customer == “jbasney@cs.wisc.edu”) ||
( Customer == “miron@cs.wisc.edu”)
10/9/2013
7
37
System Administration
Security (3)
To protect against violations of resource access policy,
The resource agent may:
Set resource consumption limit by using system API
Run the application under “guest” account
Set file system root directory to “sandbox” directory
Intercept the system calls performed by app. via OS interposition
interface
38
System Administration
Security (4)
To protect the customer’s account and files
HTC must ensure that all resource agents are trustworthy
Placing data files only on trusty hosts
Using authentication mechanism
Encrypting network streams
39
System Administration
Remote Customers
Remote access is more convenient than direct access
Customer creates an HTC account
Customer agent can be installed on customer workstation
The administrator allows this agent to access the HTC cluster
For non- trustworthy customers, extra security procedures may be
required
40
Condor
41
Condor is suitable for high throughput computations
Running many jobs at same time at different machines
Exploiting idle machines
Allowing for many jobs to be completed over a long period of
time
Useful for researchers that concern with number of jobs they
can do over particular time length
Condor
42
Running programs unattended and in the background
Redirecting console input & outputs from and to files
Notifying on completion via email
Allowing tracking jobs’ progress
Running one job on multiple machines
Survive hardware and software failure
Allowing joining and leaving of machines
Enforcing your own policy
10/9/2013
8
Condor
43
Condor can be seen as a distributed job scheduler
Scheduling submitted jobs on available machines
Allowing users to specify priorities to their jobs
Ensuring fair resources share by constantly calculating user priority
Lower numerical value means higher priority
Each user starts with the highest priority (0.5)
Priority improves over time if number of used machines < priority
Priority worsens over time if number of used machines > priority
Using of checkpoints
Suspending and resuming of jobs
Rescheduling jobs on different machines
44
Condor as Distributed Job Scheduler
Distributed Job Scheduler
45
Machine expresses
Attributes
Conditions
Preferences
Job expresses
Attributes
Requirements
Preferences
Matchmaker
Finds matching
Notifies the matched parties
Distributed Job Scheduler
ClassAd Language
Describing jobs, workstations, and other resources
Same idea of classified advertising section of news paper
Exchangeable between
processes to schedule jobs
Providing information about
the state of the system
46
Distributed Job Scheduler
ClassAd Structure
Set of attribute-values pairs
Each value can be:
Integer
Floating point
String
Logical expression
TRUE
FALSE
UNDEFINED
47
Distributed Job Scheduler
ClassAd Example:
48
Also, attributes from different ClassAds can be used
For Example: other.size > 3
10/9/2013
9
Distributed Job Scheduler
Matchmaker:
Its job to find matching between two ClassAds (job & machine)
Matching between two ClassAds (job & machine) occurs if
The expressions of Requirements attribute in both ClassAds are true
If more than two matches are found?
Rank is used
49
Distributed Job Scheduler
50
51
Condor: How to submit the job
Distributed Job Scheduler Job Submission Can be done by submit a job description file Job description file It is plain ASCII text file used to describe job or cluster (several jobs)
Specify how many times to run the job
Specify the directory of the input and output files
Specify how to receive notification when completing the execution (email or log)
Select an Universe
Standard or Vanilla
PVM
MPI
GLOBUS (Grid applications)
Scheduler (meta-schedulers)
52
53
Description file Example:
Distributed Job Scheduler
54
Description file Example:
Distributed Job Scheduler
10/9/2013
10
55
Description file Example:
Distributed Job Scheduler Distributed Job Scheduler
Standard Universe
Running serial jobs
Not supporting Checkpoint at kernel level
Relinking source code with Condor system call library
Transparently processing & restarting checkpoint
Transparently processing migration
Automatically using remote access mechanism
By default, storing checkpoint on local disk of submit machine
Configurable, it can be stored on checkpoint server
56
Distributed Job Scheduler
Standard Universe
Remote file access
57
Distributed Job Scheduler
Vanilla Universe
Running almost all serial jobs
Running any program that can run outside of Condor
Typically relying on shared file system between submit
machine and other nodes
If no shared file system, files will be transferred
58
Distributed Job Scheduler
MPI Universe
Managing parallel programs written using MPI
Uses only dedicated resources
59
Distributed Job Scheduler
PVM Universe
Giving the ability to submit PVM applications
PVM can ask Condor to add new machine
60
10/9/2013
11
61
Condor: dependencies between jobs
Distributed Job Scheduler
62
DAGMan Scheduler Using directed acyclic graph(DAG) to specify dependencies
Distributed Job Scheduler
63
DAGMan Scheduler Managing the submissions of jobs
Distributed Job Scheduler
64
DAGMan Scheduler Managing the submissions of jobs
Distributed Job Scheduler
65
DAGMan Scheduler Managing the submissions of jobs
Distributed Job Scheduler
66
DAGMan Scheduler Managing the submissions of jobs
10/9/2013
12
Condor Architecture
67
Condor Pool? Pool owns central manager and a collection of jobs and machines
Central manager serves as centralized repository of info about the state of the pool
Condor Architecture
68
Job Startup
Condor Architecture
69
INFN Condor pool
70
Grid overview What is Grid?
“Flexible, secure, coordinated resource sharing among dynamic
collections of individuals, institutions, and resources”
Virtual Organization (VO)
Dynamic Set of individuals and institutions defined by sharing rules to share their resources to achieve a common goal
Example of VO A crisis management team, the databases, and simulation systems are used to plan a response to an emergency situation
71
Grid overview
72
10/9/2013
13
Grid overview VO requirements
Flexible sharing relationships
Control on shared resources
Usage modes
Shared infrastructure services
Interoperability
Since Grid technology provides a general resource-sharing framework, it can be used to address the VO requirements
73
Grid Architecture Grid architecture must be formed as layers with hourglass shape
Each layer contains
component sharing
the same role
Component in each
layer can use services
of lower layers
The interacting between
components can be done
through standard protocols
74
Grid Architecture Fabric
Interface to local control
Implement the local, resource–specific operations
Implement resources Enquiry and resource management mechanisms to have the capability
Computational: monitoring and controlling process execution
Storage: read and writes files
Network: have control over network resources
Code Repository: managing versioned source code
75
Grid Architecture
76
Connectivity
Defines core communication and authentication protocols.
To exchange data between Fabric layer resources.
Authentication solutions :
logon once and have access to multiple Grid resources
Delegation
User-based trust relationships
Grid Architecture
77
Resource
Sharing Single Resources
Defines protocols for secure negotiation, initiation, monitoring, control, accounting, payment of sharing operations on individual resources.
Two primary classes of Resource layer protocols
Information protocols
To provide information about the structure and state of a resource
Management protocols
To negotiate access to a shared resource
Grid Architecture
78
Collective
Coordinating Multiple Resources
Defines protocols that capture interactions across collections of resources.
Service examples: Directory services Co-allocation, scheduling, and brokering services Monitoring and diagnostics services Data replication services Grid-enabled programming systems Workload management systems and collaboration frameworks Software discovery services Community authorization servers
10/9/2013
14
Grid Architecture
79
Application
Implements the business logic
Operate within VO environment
Constructed by calling services defined at any layer.
80
81
Globus Toolkit
Globus
Community of organizations and individuals developing fundamental
technologies behind the Grid
Globus Toolkit
Open source software toolkit provides basic infrastructure, protocols,
and services to build grids and applications
82
Globus Toolkit
Who is involved in Globus Alliance?
Argonne National Laboratory’s Mathematics and Computer Science Division
The University of Southern California’s Information Sciences Institute
The University of Chicago's Distributed Systems Laboratory
The University of Edinburgh in Scotland
The Swedish Center for Parallel Computers
National Computational Science Alliance
The NASA Information Power Grid project
…..
83
Globus Toolkit
Projects using Globus Toolkit
Computer Science
Condor
DOE e-Services
GridLab
GriPhyN
NMI GridShib
NMI Performance Monitoring
OGCE
OGSA-DAI
SciDAC CoG
SciDAC Data Grid
SciDAC Security
vGRADS
Physics
FusionGrid
LIGO
Particle Physics Data Grid
Infrastructure
ASCI (HPSS)
EGEE
Grid3
GRIDS Center
iVDGL
NorduGrid
Open Science Grid
TeraGrid
UK e-Science
Astronomy
Sloan Digital Sky Survey
National Virtual
Observatory
Chemistry
CMCS
Civil Engineering
NEES
Climate Studies
LEAD
Earth System Grid
Collaboration
Access Grid
84
Globus Toolkit
The Toolkit
Include a set of services and software components to
support building Grids and their applications
Includes a set of modules
Each module provides an interface used by higher-
level services to invoke the module’s mechanisms.
Each module provides implementations that use low-level operations to give the ability to implement these mechanisms in different environments
10/9/2013
15
85
Globus Toolkit
86
Globus Toolkit
Fabric:
Any resources that can be shared.
For example: Distributed file system and condor
Resources defined by vendor-supplied interfaces
Includes enquiry software to detect
resources capabilities and
delivers these information
to higher lever services
87
Globus Toolkit
Connectivity
Grid Security Infrastructure (GSI) Nexsus
88
Globus Toolkit
Resource
Grid Resource Access Management (GRAM) Grid Resource Information Protocol (GRIP) Grid Resource Registration Protocol (GRRP) GridFTP
89
Globus Toolkit
Connectivity
Grid Information Index Servers (GIISs) LDAP information protocol Dynamically Updated Request Online Co-allocator (DUROC)
90
Commonalities & Contrast
Commonalities
Using dedicated & non-dedicated resources
Providing powerful capacity
Contrast
Globus provides tools to to build girds, while Condor is software
that exploits resources of workstations to perform extensive tasks
Condor and Globus complementary technologies
Condor-G, a Globus-enabled version of Condor
10/9/2013
16
91
Inefficiencies & Possible
Problems in Condor:
One central manager
One checkpoint server
Possible Solution:
For each one, we should have mirror server that can be used
in case of crashing the original server
92
Recommended