Embed Size (px)
Citation preview
VIMS HPC Tutorial
September 16th 2015
Eric J. Walter
● Understanding the available hardware and software● How to edit files / which text editor● Basic linux and shell usage● Overview of file systems and file permissions● How to use the batch system to run jobs● Compilers and creation of programs
Available HPC hardware and software
http://www.hpc.wm.edu/SciClone/Home - Wiki homepage
http://www.hpc.wm.edu/SciClone/Software - Software list (out of date)http://www.hpc.wm.edu/SciCloneTutorials/ModuleBasics - Software modules
http://www.hpc.wm.edu/SciCloneTutorials/Home - Tutorials pagehttp://www.hpc.wm.edu/SciCloneUserGuide/Home - User's guide http://www.hpc.wm.edu/SciClone/Hardware - Hardware components
http://www.hpc.wm.edu/SciClone/presentations/ClusterOverview_permissions.pdf
Recent hardware overview:
SciCloneTyphoon
typhoo
n
4 cores / node72 nodes -> 288 cores2 or 6 GB mem / coreOpteron Santa-RosaDDR Infiniband
Vortex
vortex
12 cores / node36 nodes -> 432 cores2.7 or 10.7 GB mem / coreOpteron SeoulFDR Infiniband
● SciClone has 1232 cores total● 2 Intel & 2 Opteron clusters● Typhoon uses SLES 10 – rest RHEL 6.2● Total usage for May 2015 : 74%● hurricane cluster has 2xM2075 GPUs (Fermi) ● change all passwords on Hurricane
TwisterGulfstream
Tornado Gale
Other servers for filesystems / backup etc.
Hurricane
hurrican
e
whirlwind
8 cores / node52 nodes -> 416 cores8 or 24 GB mem / coreIntel WestmereQDR Infiniband
8 cores / node12 nodes -> 96 cores6 GB mem / coreIntel WestmereQDR Infiniband
.sciclone.wm.edu
48 cores / 96 GB 32 cores / 64 GB
ice01 ice02
Opteron Magny-CoursQDR Infiniband
snow8 cores / node15 nodes -> 120 cores2 GB mem / core Opteron ShanghaiDDR/QDR Infiniband
Storm
wind
16 cores / node26 nodes -> 416 cores2 GB mem / coreOpteron Magny-CoursQDR Infiniband
Storm
● Storm has 1080 cores total● All Opteron● RHEL 6.2● Total usage for May 2015 : 67%● Has Lustre filesystem
gust
Other servers for filesystems / backup etc.
4 cores / node98 nodes -> 392 cores2-8 GB mem / coreOpteron Santa-RosaDDR Infiniband
rain
hail8 cores / node9 nodes -> 72 cores2-8 GB mem / coreOpteron ShanghaiQDR Infiniband
.hpc.wm.edu
Chesapeake
Chesapeake
● Chesapeake has 380 cores total● All Opteron● RHEL 6.2● Total usage for May 2015 : 32% ● Still need to incorporate Indian nodes
Chickahominy
York
Rappahannock
Other servers for filesystems / backup etc.
potomac
12 cores / node30 nodes -> 360 cores2.7 GB mem / coreOpteron SeoulQDR Infiniband
in1 in2
.hpc.vims.edu
Wiki
URL: www.hpc.wm.edu/SciClone/Home
Software
FACT: Software is requested and installed more quickly than documentation can be writtenSubmit a ticket for any software requests. Please make sure you need the software.
Linux Text Editors
Popular text editors: emacs or vi/vim
emacs: huge, bloated, not installed by default, but the champion!
vi/vim: tiny, always available, some users love it
nano: editor with training wheels, very easy to use, not very powerful
http://www.gnu.org/software/emacs/tour/http://www.jesshamrick.com/2012/09/10/absolute-beginners-guide-to-emacs/http://www2.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html
Emacs Tutorials
http://www.vim.org/http://vim.wikia.com/wiki/Tutorialhttp://linuxconfig.org/vim-tutorial
Vim Tutorials
http://www.nano-editor.org/Nano homepage
Linux Shell Usage
Tutorials page – Subcluster Essentials pages ; Using the XX cluster Guides to running on a particular subcluster
http://www.hpc.wm.edu/SciCloneTutorials/LinuxUnix - page of linux tutorials
Really two goals for linux understanding
1) learning linux commands and concepts2) learning how to configure your evironment
Questions to ask yourself:
- what sofware do I want loaded by default?- what should my default umask be?- what ISA am I compiling for?- where can I store files?- where should I run my jobs?
HPC Filesystems / Backup146 [hurricane] df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/VolGroup00-LogVol00 7.9G 4.8G 2.8G 64% /tmpfs 7.8G 76K 7.8G 1% /dev/shm/dev/sda1 485M 76M 384M 17% /boot/dev/mapper/VolGroup00-LogVol03 7.9G 147M 7.4G 2% /tmp/dev/mapper/VolGroup00-LogVol02 14G 5.5G 7.7G 42% /usr/dev/mapper/VolGroup00-LogVol04 7.9G 1.7G 5.9G 23% /var/dev/mapper/VolGroup00-LogVol06 413G 7.0G 385G 2% /sciclone/scr01/dev/mapper/VolGroup30-LogVol31 917G 482G 390G 56% /sciclone/home00tn00:/usr/local 134G 113G 15G 89% /usr/localtn00:/export 46G 15G 30G 33% /importmh00:/var/spool/mail 7.9G 4.3G 3.3G 57% /var/spool/mailgfs00:/sciclone/home04 591G 429G 157G 74% /sciclone/home04ty00:/sciclone/scr02 273G 81M 273G 1% /sciclone/scr02tn00:/sciclone/scr10 7.9G 2.3G 5.3G 31% /sciclone/scr10tn00:/sciclone/scr30 17T 13T 4.7T 72% /sciclone/scr30gfs00:/sciclone/data10 16T 15T 1.9T 89% /sciclone/data10gfs00:/sciclone/vims20 26T 19T 6.7T 74% /sciclone/vims20tw00-i8:/sciclone/data20 73T 57T 16T 79% /sciclone/data20/dev/md1 8.1T 5.0T 2.8T 65% /sciclone/scr20vx00:/sciclone/home10 2.7T 202M 2.6T 1% /sciclone/home10vx00:/sciclone/scr00 318G 4.8G 297G 2% /sciclone/scr00
home – nightly backup
data – weekly backup
scr – not backed up
Sharing Files & Folders : Permissions Isee
https://www.nersc.gov/users/storage-and-file-systems/unix-file-permissionsfor more information
35 [hurricane] ls -l resultstotal 3-rw------- 1 ewalter hpcf 194 Jun 16 14:37 ww.dat-rw------- 1 ewalter hpcf 194 Jun 16 14:37 yy.dat-rw------- 1 ewalter hpcf 194 Jun 16 14:37 zz.dat
list files in directory
user group other
r-readw-writex-execute
**directories need to be ‘x’ to be entered/passed through
directo
ry
associateduser
associatedgroup
print working directory
33 [hurricane] pwd/sciclone/home04/ewalter
list directory34 [hurricane] ls -ld resultsdrwx------ 2 ewalter hpcf 512 Jun 16 14:37 results
http://www.hpc.wm.edu/SciClone/presentations/Permissions.pdf
Sharing Files & Folders : Permissions III want to share my results directory and files with those in the seadas group:
Am I in the seadas group?:
Change the group for the directory and all files below:
51 [hurricane] chmod g+rX -R results/
52 [hurricane] ls -ld results/drwxr-x--- 2 ewalter seadas 512 Jun 16 14:37 results/53 [hurricane] ls -l results/total 3-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat
Make: files group readable folders group readable and executable
46 [hurricane] groups ewalterewalter : hpcf wmall hpcstaff www seadas vasp wm sysadmin hpcadmin hpsmh
47 [hurricane] chgrp -R seadas results/
49 [hurricane] ls -ld results/drwx------ 2 ewalter seadas 512 Jun 16 14:37 results/
50 [hurricane] ls -l results/total 3-rw------- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw------- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw------- 1 ewalter seadas 194 Jun 16 14:37 zz.dat
Sharing Files & Folders : Permissions IIIWhat happens when a new file is created?
78 [hurricane] cd results/
80 [hurricane] touch newfile81 [hurricane] ls -ltotal 3-rw------- 1 ewalter hpcf 0 Jun 16 15:13 newfile-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat
This is bad since the new file isn’t in the seadas group!
88 [hurricane] chmod g+s results/89 [hurricane] cd results/
91 [hurricane] touch newfile92 [hurricane] ls -ltotal 3-rw------- 1 ewalter seadas 0 Jun 16 15:14 newfile-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 ww.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 yy.dat-rw-r----- 1 ewalter seadas 194 Jun 16 14:37 zz.dat
add “setgid” to folder only
file group inherits the group from the folder it is in
Sharing Files & Folders : Permissions IVThe user’s umask controls what permissions files and folders are given when created:
93 [hurricane] umask77
HPC default umask is 077Can be changed in your startup file (.cshrc etc.)
135 [hurricane] touch file077136 [hurricane] ls -l file077 -rw------- 1 ewalter seadas 0 Jun 16 15:24 file077
137 [hurricane] umask 022138 [hurricane] touch file022139 [hurricane] ls -l file0*-rw-r--r-- 1 ewalter seadas 0 Jun 16 15:24 file022-rw------- 1 ewalter seadas 0 Jun 16 15:24 file077
140 [hurricane] umask 027141 [hurricane] touch file027142 [hurricane] ls -l file0*-rw-r--r-- 1 ewalter seadas 0 Jun 16 15:24 file022-rw-r----- 1 ewalter seadas 0 Jun 16 15:24 file027-rw------- 1 ewalter seadas 0 Jun 16 15:24 file077
Common Permissions Tasks IChange the permissions of a directory chmod:
Change the permissions of a directory and everything under it chmod -R:
24 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP25 [hurricane] chmod go+rX VASP26 [hurricane] ls -ld VASPdrwxr-xr-x 4 ewalter hpcf 512 Apr 4 2014 VASP27 [hurricane] chmod o-rX VASP28 [hurricane] ls -ld VASPdrwxr-x--- 4 ewalter hpcf 512 Apr 4 2014 VASP29 [hurricane] chmod g-rX VASP30 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP
32 [hurricane] ls -ld VASPdrwx------ 4 ewalter hpcf 512 Apr 4 2014 VASP33 [hurricane] ls -l VASPtotal 52457-rw-r----- 1 ewalter hpcf 22932904 Apr 4 2014 potpaw_LDA.52.tar.gz-rw-r----- 1 ewalter hpcf 25958479 Apr 4 2014 potpaw_PBE.52.tar.gzdrwxr-x--- 2 ewalter hpcf 15360 Aug 6 00:01 vasp.5.3
34 [hurricane] chmod -R g+rX VASP
35 [hurricane] ls -ld VASPdrwxr-x--- 4 ewalter hpcf 512 Apr 4 2014 VASP36 [hurricane] ls -l VASPtotal 52457-rw-r----- 1 ewalter hpcf 22932904 Apr 4 2014 potpaw_LDA.52.tar.gz-rw-r----- 1 ewalter hpcf 25958479 Apr 4 2014 potpaw_PBE.52.tar.gzdrwxr-x--- 2 ewalter hpcf 15360 Aug 6 00:01 vasp.5.3
Use this command if you want to allow group access to your home, scrXX, and dataXX directories
HPC default umask is 077
How do I change the initial permissions that files and folders are given when created:
You need to edit your .cshrc file in your home directory and add: umaskumask 077 files get “-rw-------” folders get “drwx------” umask 027 files get “-rw-r-----” folders get “drwxr-x---” umask 022 files get “-rw-r--r--” folders get “drwxr-xr-x”
What groups are I in?: groups52 [hurricane] groups ewalterewalter : hpcf wmall hpcstaff www seadas vasp sysadmin wm hpcadmin wheel hpsmh
My primary (default) group is hpcf and the rest are secondary
Change a group associated with a file or directory: chgrp 54 [hurricane] ls -ld projectdrwx------ 2 ewalter hpcf 512 Aug 18 20:47 project55 [hurricane] chgrp hpcstaff project56 [hurricane] ls -ld projectdrwx------ 2 ewalter hpcstaff 512 Aug 18 20:47 project
Common Permissions Tasks II
Using the Batch Systemhttp://www.hpc.wm.edu/SciCloneTutorials/LaunchJobs
Concepts:Interactive vs. Batch jobSubmitting / checking / deleting Forwarding X
Special VIMS commands:qsubdep – submit a new job with a dependency on all current jobsqpri – arrange a list of jobs to run in a particular order, one at a time
13 [chesapeake] qstat u ewalter (qsu)
Job id Name User Time Use S Queue 1000.cp00 Job1 ewalter 00:00:29 R qx 1001.cp00 Job2 ewalter 31:56:22 R qx 1002.cp00 Job3 ewalter 0 Q qx
14 [chesapeake] qsubdep runscript
Job id Name User Time Use S Queue 1000.cp00 Job1 ewalter 00:00:29 R qx 1001.cp00 Job2 ewalter 31:56:22 R qx 1002.cp00 Job3 ewalter 0 Q qx1003.cp00 Job4 ewalter 0 H qx
Using the Batch System13 [chesapeake] qsub h run1 13 [chesapeake] qsub h run2 13 [chesapeake] qsub h run3
13 [chesapeake] qstat u ewalter
Job id Name User Time Use S Queue 1000.cp00 Job1 ewalter 0 H qx 1001.cp00 Job2 ewalter 0 H qx 1002.cp00 Job3 ewalter 0 H qx
14 [chesapeake] qpri r 1000 1001 1002
Job id Name User Time Use S Queue 1000.cp00 Job1 ewalter 00:00:00 R qx 1001.cp00 Job2 ewalter 0 H qx 1002.cp00 Job3 ewalter 0 H qx
qpri - hold each job until the previous finishes the -r flag removes the user hold on the jobs if you skip -r, you need to use qrls to release the hold qrls <jobid>
CompilersAll platforms have GNU and Portland C/C++/FORTRAN compilersCheck the Tutorials - Subcluster Essentials / Using XX pages for what flags are suggested for optimization.
For Chesapeake / Potomac:
Important to know what the isa of the target processor is. All potomac nodes are type seoul
