Embed Size (px)
Citation preview
Small Tutorial on HTCondor
Claudia Solís-Lemus1
March 23, 2015
1based on slides by Steve Hunter and Lauren Michael
HTCondor: in its simplest form
CSL HTCondor March 23, 2015 2 / 33
When to use HTCondor?
Many independent jobs to run in parallel: high-throughput computing(HTC):
your computing work won’t run at all on your computer(s) (lacksufficient RAM, disk,...)your computing work will take too long on your own computer(s)simply would like to off-load certain processes in favor of runningothers on your computer(s)
Ideal running time for a single job: between 5 minutes and 2 hours -but CHTC pool allows up to 72h (even longer jobs possible)
Keep in mind: high-performance computing (HPC) also available inCHTC: http://chtc.cs.wisc.edu/HPCuseguide.shtml
CSL HTCondor March 23, 2015 3 / 33
When to use HTCondor?
Many independent jobs to run in parallel: high-throughput computing(HTC):
your computing work won’t run at all on your computer(s) (lacksufficient RAM, disk,...)your computing work will take too long on your own computer(s)simply would like to off-load certain processes in favor of runningothers on your computer(s)
Ideal running time for a single job: between 5 minutes and 2 hours -but CHTC pool allows up to 72h (even longer jobs possible)
Keep in mind: high-performance computing (HPC) also available inCHTC: http://chtc.cs.wisc.edu/HPCuseguide.shtml
CSL HTCondor March 23, 2015 3 / 33
When to use HTCondor?
Many independent jobs to run in parallel: high-throughput computing(HTC):
your computing work won’t run at all on your computer(s) (lacksufficient RAM, disk,...)your computing work will take too long on your own computer(s)simply would like to off-load certain processes in favor of runningothers on your computer(s)
Ideal running time for a single job: between 5 minutes and 2 hours -but CHTC pool allows up to 72h (even longer jobs possible)
Keep in mind: high-performance computing (HPC) also available inCHTC: http://chtc.cs.wisc.edu/HPCuseguide.shtml
CSL HTCondor March 23, 2015 3 / 33
When to use HTCondor?
Many independent jobs to run in parallel: high-throughput computing(HTC):
your computing work won’t run at all on your computer(s) (lacksufficient RAM, disk,...)your computing work will take too long on your own computer(s)simply would like to off-load certain processes in favor of runningothers on your computer(s)
Ideal running time for a single job: between 5 minutes and 2 hours -but CHTC pool allows up to 72h (even longer jobs possible)
Keep in mind: high-performance computing (HPC) also available inCHTC: http://chtc.cs.wisc.edu/HPCuseguide.shtml
CSL HTCondor March 23, 2015 3 / 33
When to use HTCondor?
Many independent jobs to run in parallel: high-throughput computing(HTC):
your computing work won’t run at all on your computer(s) (lacksufficient RAM, disk,...)your computing work will take too long on your own computer(s)simply would like to off-load certain processes in favor of runningothers on your computer(s)
Ideal running time for a single job: between 5 minutes and 2 hours -but CHTC pool allows up to 72h (even longer jobs possible)
Keep in mind: high-performance computing (HPC) also available inCHTC: http://chtc.cs.wisc.edu/HPCuseguide.shtml
CSL HTCondor March 23, 2015 3 / 33
Real-life example in HTCondor: phylogenetics
Run 1800 quartets, eachwith 4000 genesEach gene: 2 minutesEach quartet: 5.5 daysAll quartets: 30 yearsWith HTCondor: 9 days
CSL HTCondor March 23, 2015 4 / 33
HTCondor: in its simplest form
Job Computer
HTCondor
CSL HTCondor March 23, 2015 5 / 33
HTCondor: in its simplest form
Job Computer
Matchmaker
CSL HTCondor March 23, 2015 6 / 33
How does HTCondor know what you want/need?
Submit file
CSL HTCondor March 23, 2015 7 / 33
How does HTCondor know what you want/need?
Submit file
CSL HTCondor March 23, 2015 7 / 33
Simple submit file
CSL HTCondor March 23, 2015 8 / 33
Not so simple submit file
CSL HTCondor March 23, 2015 9 / 33
Not so simple submit file
kbMb
CSL HTCondor March 23, 2015 10 / 33
Another submit file
CSL HTCondor March 23, 2015 11 / 33
Submit to HTCondor
CSL HTCondor March 23, 2015 12 / 33
Check the queue in HTCondor
CSL HTCondor March 23, 2015 13 / 33
Submit to HTCondor: commands
condor_submit submitfilecondor_q usernamecondor_rm jobID or username
CSL HTCondor March 23, 2015 14 / 33
Submit to HTCondor
Submit
nodeExecute
node
HTCondor
desk22 (where job runs)
CSL HTCondor March 23, 2015 15 / 33
CHTC pools
CSL HTCondor March 23, 2015 16 / 33
CHTC pools
simon → Mike Camilleri [email protected]
CHTC → chtc.cs.wisc.edu, click "Get Started"
CSL HTCondor March 23, 2015 17 / 33
CHTC pools
simon → Mike Camilleri [email protected]
CHTC → chtc.cs.wisc.edu, click "Get Started"
CSL HTCondor March 23, 2015 17 / 33
Checklist
X Input filesX My codeX Submit fileX condor_submit submitfileX Monitor/remove jobs
Is it that simple?
CSL HTCondor March 23, 2015 18 / 33
Checklist
X Input filesX My codeX Submit fileX condor_submit submitfileX Monitor/remove jobs
Is it that simple?
CSL HTCondor March 23, 2015 18 / 33
Code in:
C++Java
MatlabR
CSL HTCondor March 23, 2015 19 / 33
Code in:
C++Java
MatlabR
CSL HTCondor March 23, 2015 19 / 33
chtc.cs.wisc.edu
CSL HTCondor March 23, 2015 20 / 33
chtc.cs.wisc.edu
CSL HTCondor March 23, 2015 21 / 33
Steps to run R code in HTCondor
1 Create R script2 Take your R version and libraries with you3 Download the ChtcRun package and follow theinstructions
4 Run mkdag and submit jobs
CSL HTCondor March 23, 2015 22 / 33
Steps to run R code in HTCondor
Step 1: R script
Create your R script
Debug and test thoroughly locally
Organize your input files
Keep in mind what output files are you expecting
Warning: no folder structure in execute node
CSL HTCondor March 23, 2015 23 / 33
Steps to run R code in HTCondor
Step 2: R version,libraries
R version (list of options in CHTC website)List of needed R packages (order important): copy source .tar.gz filesYou need to be in a CHTC (or simon) submit node
CSL HTCondor March 23, 2015 24 / 33
Steps to run R code in HTCondor
Step 3: ChtcRun package
CSL HTCondor March 23, 2015 25 / 33
Steps to run R code in HTCondor
Step 3: ChtcRun package
Organize your input files as:ChtcRun/
Rin/0/ infile.txt, <specific files>1/ infile.txt, <specific files>job2/ infile.txt, <specific files>shared/ RLIBS.tar.gz, yourScript.r, <shared files>
CSL HTCondor March 23, 2015 26 / 33
Steps to run R code in HTCondor
Step 3: ChtcRun package
Modify process.template with:
request memory, request disk (make small test first)
wantFlocking? wantGlidein?
CSL HTCondor March 23, 2015 27 / 33
Steps to run R code in HTCondor
Step 4: Run mkdag
Run ./mkdag with options
See ./mkdag --help for examples
./mkdag --data=Rin --outputdir=Rout --cmdtorun=soartest.R--pattern=meanx --type=R --version=R-2.10.1
Message: All done!
CSL HTCondor March 23, 2015 28 / 33
Steps to run R code in HTCondor
Step 4: Run mkdag
Run ./mkdag with options
See ./mkdag --help for examples
./mkdag --data=Rin --outputdir=Rout --cmdtorun=soartest.R--pattern=meanx --type=R --version=R-2.10.1
Message: All done!
CSL HTCondor March 23, 2015 28 / 33
Steps to run R code in HTCondor
Step 4: Submit your jobs
mkdag will instruct you:
cd Routcondor_submit_dag mydag.dag
Don’t forget to monitor your jobs: condor_q and check log files formemory/disk requirements
CSL HTCondor March 23, 2015 29 / 33
Steps to run R code in HTCondor
Step 4: Submit your jobs
mkdag will instruct you:
cd Routcondor_submit_dag mydag.dag
Don’t forget to monitor your jobs: condor_q and check log files formemory/disk requirements
CSL HTCondor March 23, 2015 29 / 33
Important things to remember!
Test your code locallyThere is no folder structure in execute nodeTest into HTCondor gradually: 3 jobs, 100 jobs, 1000 jobs...Adjust memory/disk requirementsAvoid transferring large files through submit node (> 10GB perbatch, ∼ 10MB per job, 1000 jobs
CSL HTCondor March 23, 2015 30 / 33
Workflow - HTCondor DAGman
http://research.cs.wisc.edu/htcondor/dagman/dagman.html
CSL HTCondor March 23, 2015 31 / 33
Workflow - HTCondor DAGman
http://research.cs.wisc.edu/htcondor/dagman/dagman.html
CSL HTCondor March 23, 2015 31 / 33
Help resources
chtc.cs.wisc.edu
research.cs.wisc.edu/htcondor/tutorials/
CHTC office hours: Wednesday 9:30-11:30am
CSL HTCondor March 23, 2015 32 / 33
chtc.cs.wisc.edu
CSL HTCondor March 23, 2015 33 / 33
