Upload
maeve-pinnock
View
215
Download
0
Embed Size (px)
Citation preview
SARA Reken- en Netwerkdiensten ToPoS | 3 juni 2007
More efficient job submission
Evert Lammerts
SARA Computing and Networking Services
High Performance Computing and Visualization
e-Science Support
SARA Reken- en Netwerkdiensten
Job submission works!
But it’s up to YOU to monitor the cycle of a jobstrange return codes
does the job status (Success, Failed) really mean what you think it does
resubmit failed jobs
Etc, etc, etc…
Success rate: 70%
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Pilot Jobs I
Normally:
With Pilot jobs:
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Pilot Jobs II
Pilot jobs:Are basically loops
Find out what to do by fetching tasks
Run till there are no more tasks, or…
Run till they are out of Wall Time
Tasks:Contain parameters
Are served by ToPoS from a webserver
When there are no more tasks, you’re sure that you’re done!
ToPoS| 3 juni 2007
SARA Reken- en Netwerkdiensten
ToPoS
is a service
is based on HTTP
can be queried by any HTTP client!
is strong because it’s simple!
Has task realms and pools
https://topos.grid.sara.nl/3/
ToPos| 3 juni 2007
SARA Reken- en Netwerkdiensten
Great! But now what…
A trivial example: matrix multiplication
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Approach
1 file per row of matrix A
1 file per column of matrix B
The Task:Contains an A[row] and B[col]
Is a shell script that can be sourced
The Pilot job:Fetches a task (None? Exit!)
Sources the task
Calculates the result
Puts the result in a file ([row]-[col].txt)
Deletes the task
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Upload files to UI
Wget https://hpcv.projects.sara.nl/wiki/images/0/06/ToposDemo.zip
WinSCP?
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Lets go! I
Login to UIssh [user]@gb-se-ams.els.sara.nl (or putty!)
Create proxystartGridSession [VO]
Enter certificate password
Unpack the demo:tar -xvvf toposDemo.tar.gz
Unzip -xvvf toposDemo.zip
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Transfer input data
export LFC_HOST='lfc.grid.sara.nl‘
lfc-mkdir /grid/[VO]/[USERNAME]
inputdata/copyFiles.sh:
#!/bin/bash
if [ $# -ne 2 ]
then
echo "Supply your VO name and your username!"
exit 0
fi
for x in *.txt
do
lcg-cr --vo ${1} -d srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/${1}/${2}/${x} -l lfn:/grid/${1}/${2}/${x} "file://${PWD}/${x}"
done
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Pilot Job
Pilotjob.sh
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
JDL
pilotjob.jdl:
Type = "Job";
JobType = "Normal";
Executable = "/bin/sh";
Arguments = "pilotjob.sh [REALM_URL] [VO] [USER]";
StdOutput = "stdout.log";
StdError = "stderr.log";
InputSandbox = {"pilotjob.sh"};
OutputSandbox = {"stdout.log","stderr.log"};
ShallowRetryCount = 1;
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Submit 10 pilot jobs
submit.sh:
#!/bin/bash
touch jobids.txt
for i in {1..10}do glite-wms-job-submit –d $USER pilotjob.jdldone
ToPoS | 3 juni 2007
SARA Reken- en Netwerkdiensten
Check status
glite-wms-job-status --input jobids.txt
ToPoS | 3 juni 2007