Embed Size (px)
Citation preview
IntrotoMonsoonandSlurm
1/18/2017
Introductions
• Introduceyourself– Name– Department/Group– Whatproject(s)doyouplantousemonsoonfor?– LinuxorUnixexperience– Previousclusterexperience?
ListofTopics
• Clustereducation– Whatisacluster,exactly?– Queues,schedulingandresourcemanagement
• ClusterOrientation– Monsoonclusterspecifics– HowdoIusethiscluster?– Groupresourcelimits– Exercises– Questionandanswer
Whatisacluster?
• Acomputerclusterismanyindividualcomputerssystems(nodes)networkedtogetherlocallytoserveasasingleresource
• Abilitytosolveproblemsonalargescalenotfeasiblealone
InsideaNode
Socket0
Socket3Socket2
Socket1
Whatisaqueue?
• Normallythoughtofasaline,FIFO• QueuesonaclustercanbeasbasicasaFIFO,orfarmoreadvancedwithdynamicprioritiestakingintoconsiderationmanyfactors
Whatisscheduling?
• “Aplanorprocedurewithagoalofcompletingsomeobjectivewithinsometimeframe”
• Schedulingforaclusteratthebasiclevelismuchthesame. Assigningworktocomputerstocompleteobjectiveswithinsometimeavailability
• Notexactlythateasythough. Manyfactorscomeintoplayschedulingworkonacluster.
Scheduling
• A schedulerneedstoknowwhatresourcesareavailableonthecluster
• Assignmentofworkonaclusteriscarriedoutmostefficientlywithschedulingandresourcemanagementworkingtogether
ResourceManagement
• Monitoringresourceavailabilityandhealth• Allocationofresources• Executionofresources• Accountingofresources
Clusterschedulinggoals
• Optimizequantityofwork• Optimizeusageofresources• Serviceallusersandprojectsjustly• Makeschedulingdecisionstransparent
ClusterResources
• Node• Memory• CPU’s• GPU’s• Licenses
Manyschedulingmethods
• FIFO– Simplyfirstinfirstout
• Backfill– Runssmallerjobswithlowerresourcerequirementswhilelargerjobswaitforhigherresourcerequirementstobeavailable
• Fairshare– Prioritizesjobsbasedonusersrecentresourceconsumption
Monsoon• TheMonsoonclusterisaresourceavailabletotheNAUresearchenterprise
• 32systems(nodes)– cn[1-32]• 884cores• 12GPUs,NVIDIATeslaK80• Red Hat EnterpriseLinux6.7• 12TBmemory - 128GB/nodemin,1.5TBmax• 170TBhigh-speed scratch storage• 500TBlong-term storage• Highspeed interconnect:FDRInfiniband
Monsoonscheduling
• Slurm (SimpleLinuxUtilityforResourceManagement)• Excellentresourcemanagerandscheduler• Precisecontroloverresourcerequests• DevelopedatLLNL,continuedbySchedMD• Usedeverywherefromsmallclusterstothelargestclusters:
– SunwayTaihuLight (#1),10.6Mcores,93PF,15kKW- China– Titan(#3),561Kcores,17.6PF,8kKW,UnitedStates
SmallCluster!
Dualcore?
LargestCluster!
10,649,600 cores
Monsoonscheduling
• Combinationofschedulingmethods• Currentlyconfiguredtoutilizebackfillalongwithamultifactorprioritysystemtoprioritizejobs
Factorsattributingtopriority
• Fairshare (predominantfactor)– Prioritypointsdeterminedonusersrecentresourceusage– Decayhalflifeover1days
• QOS(QualityofService)– SomeQOShavehigherprioritythanothers,forinstance:debug
• Age– howlonghasthejobsatpending• Jobsize- thenumberofnodes/cpus ajobisrequesting
Storage
• /home– 10GBquota– Keepyourscriptsandexecutables here– Snapshottedtwiceaday:/home/.snapshot– Pleasedonotwritejoboutput(logs,results)here!!
• /scratch– 30dayretention– Veryfaststorage,capableof11GB/sec– Checkpoints,logs– Keepalltemp/intermediatedatahere– Shouldbeyourdefaultlocationtoperforminput/output
Storage
• /projects– Long-termstorageprojectshares– Spaceisassignedtofacultymemberforgrouptoshare– Snapshotsavailable– Nobackupstoday
• /common– Clustersupportshare– Contrib:placetoputscripts/libs/confs/db’s forothersuse
DataFlow
1. Keepscriptsandexecutablesin/home2. Writetemp/intermediatedatato/scratch3. Copydatato/projects/<group_project>,forgroupstorage
andreferenceinotherprojects4. Cleanup/scratch
**Remember,/scratchisascratchfilesystem,usedforhigh-speedtemporary,andintermediatedata
Remotestorageaccess
• scp– scp [email protected]:/scratch/nauid– WinSCP
• samba/cifs– \\nau.froot.nau.edu\cirrus(windows)– smb://nau.froot.nau.edu/cirrus(mac)
Groups
• NAUhasaresourcecalledEnterprisegroups• Theyareavailabletoyouontheclusterifyou’dliketomanageaccesstodata
• Manageaccesstoyourfiles• https://my.nau.edu
– “GotoEnterpriseGroups”– TakealookatourFAQ::nau.edu/hpc/faq
• Iftheyarenotworkingforyou,contactITShelpdesk
Software• ENVI/IDL• Matlab• IntelCompilers,andMKL• R• Qiime• AnacondaPython• OpenFOAM• SOWFA• Lotsofbioinformaticsprograms• Requestadditionalsoftwaretobeinstalled!
Modules
• Softwareenvironmentmanagementhandledbythemodulespackagemanagementsystem
• moduleavail– whatmodulesareavailable• modulelist– modulescurrentlyloaded• moduleload<modulename>- loadapackagemodule• moduledisplay<modulename>- detailedinformationincludingenvironmentvariableseffected
MPI
• QuicknoteonMPI• MessagePassingInterfaceforparallelcomputing• OpenMPIsetasdefaultMPI• Mvapich2alsoavailable
– moduleunloadopenmpi– moduleloadmvapich2
• ExampleMPIjobscript:– /common/contrib/examples/job_scripts/mpijob.sh
InteractingwithSlurm• Decidewhatyouneedtoaccomplish• Whatresourcesareneeded?
– 2cpus,12GBmemory,for2hours?• Whatstepsarerequired?
– Runprog1,thenprog2…etc– Arethestepsdependentononeanother?
• Canyourwork,orprojectbebrokenupintosmallerpieces?Smallerpiecescanmaketheworkloadmoreagile.
• Howlongshouldyourjobrunfor?• Isyoursoftwaremultithreaded,usesOpenMP orMPI?
ExampleJobscript• #!/bin/bash• #SBATCH--job-name=test• #SBATCH--output=/scratch/nauid/output.txt• #SBATCH--time=20:00 #shortertime=soonerstart• #SBATCH--workdir=/scratch/nauid
• #replacethismodulewithsoftwarerequiredinyourworkload• moduleloadpython/3.3.4
• #examplejobcommands• #eachsrun commandisajobstep,sothisjobwillhave2steps• srun sleep300• srun python-V
Interactive/DebugWork
• Runyourcompilesandtestingontheclusternodesby:
– srun –pallgcc hello.c –oa.out– srun –pall-c12make-j12– srun –qos=debug-c12make-j12– srun Rscript analysis.r– srun pythonanalysis.py– srun cp -av /scratch/NAUID/lots_o_files /scratch-lt/NAUID/destination
LongInteractivework• salloc
– ObtainaSLURMjoballocationthatyoucanworkwithforanextendedamountoftimeinteractively.Thisisusefulfortesting/debuggingforanextendedamountoftime.
[cbc@wind ~]$salloc –c8--time=2-00:00:00salloc:Grantedjoballocation33442[cbc@wind ~]$srun pythonanalysis.py[cbc@wind ~]$exitsalloc:Relinquising job allocation 33442[cbc@wind ~]$salloc -N2salloc:Grantedjoballocation33443[cbc@wind ~]$srun hostnamecn3.nauhpccn2.nauhpc[cbc@wind ~]$exitsalloc:Relinquising job allocation 33443
JobParametersYouwant SwitchesneededMorethan onecpu forthejob --cpus-per-task=2, or-c2
To specify an ordering of your jobs --dependency=afterok:job_id,or-djob_id
Split up the output, and errors --output=result.txt --error=error.txtTo run your job at a particular time/day
--begin=16:00 --begin=now+1hour --begin=2010-01-20T12:34:00
Add MPItasks/rankstoyourjob --ntasks=2, or-n2To control job failure options --norequeue –requeueToreceivestatusemail --mail-type=ALL
Contraints andResourcesYouwant SwitchesneededTo choose a specific node feature (e.g. avx2) --constraint=avx2
To use a generic resources (e.g. a gpu) --gres=gpu:tesla:1
To reserve a whole node for yourself --exclusive To chose a partition --partition
Submitthescript
[cbc@wind ~]$sbatch jobscript.shSubmittedbatchjob85223
– slurm returns ajob idforyour job that you can use tomonitorormodify constraints
Monitoringyourjob
• squeue– viewinformationaboutjobslocatedintheSLURMschedulingqueue.
• squeue --start• squeue -ulogin• squeue -o“%j%u…“• squeue -ppartitionname• squeue -Ssortfield• squeue -t<state>(PDorR)
Clusterinfo
• sinfo– viewinformationaboutSLURMnodesandpartitions.
• sinfo -N–l• sinfo –R
– Listreasonsfordownednodesandpartitions
Monitoringyourjob
• sprio– viewthefactorsthatcompriseajob’sschedulingpriority
• sprio –l-- listpriorityofusersjobsinpendingstate
• sprio -o“%j%u…“• sprio -w
Monitoringyourjob
• sstat– Displayvariousstatusinformationofarunningjob/step.
• sstat -jjobid• sstat -oAveCPU,AveRSS• Onlyworkswithjobscontainingsteps
Controling yourjob
• scancel– UsedtosignaljobsorjobstepsthatareunderthecontrolofSlurm.
• scancel -jjobid• scancel -njobname• scancel -umylogin• scancel -tpending(onlyyours)
Controllingyourjob
• scontrol– UsedtoviewandmodifySlurm configurationandstate.
• scontrol showjob85224
JobAccounting• sacct
– displaysaccountingdataforofyourjobsandjobstepsintheSLURMjobaccountinglogorSLURMdatabase
• sacct -jjobid -ojobid,elapsed,maxrss• sacct -Nnodelist• sacct -umylogin
• Tryouralias“jobstats”– jobstats– jobstats –j<jobid>
JobAccounting
• sshare– Toolforlistingthesharesofassociationstoacluster.
• sshare -l:viewandcompareyourfairshare withotheraccounts
• sshare -a:viewallusersfairshare• sshare –A–a<account>:viewallmembersinyouraccount(group)
Accounthierarchy• Youruseraccountbelongstoaparentfacultyaccount(group)• Youruseraccountsharesresourcesthatareprovidedforyourgroup
• Example:– coffey
• cbc• mkg52
• Viewtheaccountstructureyoubelongtowith:“sshare -a–A<account>”
• Example:– sshare -a-Acoffey
Limitsontheaccount(group)
• Limitsareinplacetopreventintentionalorunintentionalmisuseofresourcestoensurequickandfairturnaroundtimesonjobsforeveryone.
• Groupsarelimitedtoatotalnumberofcpu minutesinuseatonetime:700,000
• Thiscpu resourcelimitmechanismisreferredtoas:“TRESRunMins”
TRESRunMins Limit
• Whattheheckisthat!?• Anumberwhichlimitsthetotalnumberofremainingcpu minuteswhichyourrunning jobscanoccupy.
• Enablesflexibleresourcelimiting• Staggersjobs• Increasesclusterutilization• Leadstomoreaccurateresourcerequests
• Sumofjobs(cpus *timelimit remaining)
Examples
• 14400=10jobs,1cpu,1dayinlength• 144000=10jobs,10cpu,1dayinlength• 576000=10jobs,10cpu,5daysinlength• 648000=100jobs,1cpu,½dayinlength
Questions?
• Checkyourgroupscpu minusage:– sshare -l
Exercise1
GettoknowmonsoonandSlurm,onyourown.
1. Howmanynodesmakeupmonsoon?– Hint:use“sinfo”
2. Howmanynodesareintheallpartition?3. Howmanyjobsarecurrentlyintherunningstate?
– Hint:use“squeue -tR”4. Howmanyjobsarecurrentlyinthependingstate?Why?
– Hint:use“squeue –tPD”
Exercise2• Createasimplejobinyourhomedirectory• Examplehere:/common/contrib/examples/job_scripts/simplejob.sh (copyitifyoulikeJ )
• Nameyourjob:“exercise”• Nameyourjobsoutput:“exercise.out”• Outputshouldgoto/scratch/<user>/exercise.out• Loadthemodule“workshop”• Runthe“date”command• Andadditionally,the“secret”command• Submityourjobwithsbatch,i.e.“sbatch simplejob.sh”
Exercise3• Makeyourjobsleepfor5minutes(sleep300)
– Sleepisacommandthatcreatesaprocessthat…sleeps• Monitoryourjob
– squeue -uyour_nauid– squeue -tR– scontrol showjobjobnum– sacct -jjobnum
• Inspectthesteps
• Cancelyourjob– scancel jobnum
Exercise4• Copyjobscriptandedit:
– /common/contrib/examples/job_scripts/lazyjob.sh
• Submitthejob,itwilltake65sectocomplete• Usesstat andmonitorthejob
– sstat -j<jobid>• Reviewtheresourcesthatthejobused
– jobstats -j<jobid>• Wearelookingfor“MaxRSS”,MaxRSS isthe
maxamountofmemoryused• Editthejobscript,reducethememorybeing
requestedinMBandresubmit,edit“--mem=“,e.g.--mem=600
• Reviewtheresourcesthattheoptimizedjobutilizedonceagain– jobstats -j<jobid>
• Ok,memorylooksgood,butnoticethattheusercpuisthesameastheelapsedtime
Usercpu =num utilizedcpus *elapsedtime
• Thisisbecausetheapplicationwewererunningonlyused1ofthe4cpus thatwerequested
• Editthelazyjobscript,commentoutfirstsruncommand,anduncommentthesecondsruncommand.
• Resubmit• Rerunjobstats -j<jobid>,noticenowusercpu isa
multipletimestheelapsedtime,inthiscase(4).Becausewewereallocated4cpus,andused 4cpus.
Slurm Arrays!
Slurm ArraysExercise
• Fromyourscratchdirectory:“/scratch/nauid”• tarxvf /common/contrib/examples/bigdata_example.tar• cdbigdata• editthefile“job_array.sh”sothatitworkswithyournauidreplacingallNAUIDwithyours
• Submitthescript“sbatch job_array.sh”• Run“squeue”,noticethereare5jobsrunning,howdidthathappen!
MPIExample
• RefertotheMPIexamplehere:– /common/contrib/examples/job_scripts/mpijob.sh
• Editit,foryourworkareas,thenexperiment:– Changenumberoftasks,nodes…etc
• Alsocanruntheexamplelikethis:– srun --qos=debug–n4/common/contrib/examples/mpi/hellompi
Keepthesetipsinmind
• Knowthesoftwareyouarerunning• Requestresourcesaccurately• Supplyanaccuratetimelimitforyourjob• Don’tbelazy,itwilleffectyouandyourgroupnegatively
QuestionandAnswer
• Moreinfohere:http://nau.edu/hpc
• Linuxshellhelphere:– http://linuxcommand.org/tlcl.php– Freebookdownload
• Andonthenauhpc listserv– [email protected]
