Introduction to theLinux Command Line for
High-Performance Computing
Dr. Charles J AntonelliLSAIT ARSJune, 2014
cja 2014 2
RoadmapLinux overviewThe command shellNavigating the filesystemBasic commands & wildcardingShell redirection & pipeliningEditing text filesPermissionsProcesses
06/14
cja 2014 3
PollHow many here to learn the command line
For HPC, specifically Flux, work?For other computing cluster work?For desktops/other Linux/Unix/Mac OS X?Other?
06/14
cja 2014 4
Course TextWilliam E Shotts, Jr.,“The Linux Command Line: A Complete Introduction,”No Starch Press, January 2012.
Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf.
06/14
cja 2014 5
Linux Overview
06/14
cja 2014 6
A compute node
12 Intel cores
48 GB RAM
Local disk
Network06/14
cja 2014 7 06/14http://linuxforums.org.uk/index.php?action=media;sa=item;in=262
cja 2014 8
The command shell
06/14
cja 2014 9
The command lineA basic way of interacting with a Linux system
Execute commandsCreate files and directoriesEdit file contentAccess the webCopy files to and from other hostsRun HPC jobs
… do things you can’t do from the conventional point-and-click Graphical User Interface (GUI)
06/14
cja 2014 10
Why command line?1. Linux was designed for the command line2. You can create new Linux commands using the
command line, without programming3. Many systems provide only the command line, or
poorly support a GUI interface• Such as most HPC systems
4. Many things can be accomplished only through the command line• Much systems administration & troubleshooting
5. You want to be cool
06/14
cja 2014 11
The command shellThe command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executedWhen you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shellOn your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window
06/14
cja 2014 12
The command shell
06/14http://www.livefirelabs.com/208_global/images/unix-operating-system.gif
cja 2014 13
Types of shellsStandard Linux shell (used in this course)
bash “Bourne-again” shellOther shells
tcsh “Enhanced C shell”, better version of cshcsh “C shell”, common on older Unix systemssh Bourne shell, original AT&T Unix shell
What shell am I using?echo $SHELL
06/14
cja 2014 14
Connecting via sshTerminal emulators
Linux and Mac OS XStart TerminalUse ssh command
WindowsSSH Secure Shell (deprecated)U-M PuTTY/WinSCP (U-M Blue Disc)https://www.itcs.umich.edu/bluedisc/PuTTYhttp://www.chiark.greenend.org.uk/~sgtatham/putty/
06/14
cja 2014 15
Lab 0Task: Start a local shell on Mac OS X1. Login to Mac OS X with your uniqname and Kerberos password2. If there is a Terminal icon in the Dock , double-click it
Otherwise:a. Bring up a Finder window:
Click on the desktop, type Command-Nb. Start the Terminal Application:
In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click it
3. This creates a Terminal window with a shell running inside it4. From a Terminal window, Command-N will start a new Terminal
06/14
cja 2014 16
Lab 0
06/14
cja 2014 17
The shell promptThe “~$ “ is the shell prompt
This means the shell is waiting for you to type somethingFormat can vary, usually ends with “$” , “%” or “#”
If $ or %, you have a normal shellThis shell has your privileges
If #, you have a so-called “root shell”This shell has administrator privileges You can do a great deal of irreversible damage
06/14
cja 2014 18
Typing into the shellBasic input line editing commands
Backspace erases previous characterLeft and right arrow move insertion point on the lineControl-U erases the line so you can start overEnter executes the line you typedControl-C interrupts whatever command you started and returns you to the shell prompt (usually)Up and down arrow will access your command historyType “exit” without the quotes to exit the shellClick the red circle at upper left of the Terminal window to close it
06/14
cja 2014 19
Lab 1Task: Enter some basic commands~$ date~$ id~$ ps~$ df -kh~$ who~$ top
06/14
cja 2014 20
Interlude: Flux
06/14
cja 2014 21
FluxFlux is a university-wide shared computational discovery / high-performance computing service. Interdisciplinary
Provided by Advanced Research Computing at U-M (ARC)Operated by CAEN HPCHardware procurement, software licensing, billing support by U-M ITSUsed across campus
Collaborative since 2010Advanced Research Computing at U-M (ARC) College of Engineering’s IT Group (CAEN)Information and Technology ServicesMedical SchoolCollege of Literature, Science, and the ArtsSchool of Information
06/14
http://arc.research.umich.edu/resources-services/flux/
cja 2014 22
The Flux clusterLogin nodes Compute nodes
Storage…
Data transfernode
06/14
cja 2014 23
A Flux node
12 Intel cores
48 GB RAM
Local disk
Network06/14
cja 2014 24
Logging in to Fluxssh flux-login.engin.umich.edu
MToken (or Software Token) requiredYou will be randomly connected a Flux login node
Currently flux-login1 or flux-login2Firewalls restrict access to flux-login.To connect successfully, either
Physically connect your ssh client platform to the U-M campus wired or MWireless network, orUse VPN software on your client platform, orUse ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there
06/14
cja 2014 25
A shell on a login nodeNow you have a shell on the login nodeYour shell prompt has changed to something like[flux-login1 ~]$ Try those simple commands here!Flux is for HPC work
The login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runsFor HPC computation, you should run on the compute nodes
A batch job will always run on the compute nodesAn interactive batch job will get you a shell on a compute node
06/14
cja 2014 26
A shell on a compute node
Getting a shell on a compute nodeEnter this command on the login node:[flux-login1 ~]$ ~cja/hpc100/getshellAfter a short delay, you’ll get an interactive shell:qsub: waiting for job 11147004.nyx.engin.umich.edu to startqsub: job 11147004.nyx.engin.umich.edu ready[nyx5509 ~]$ You can type commands into this shell for all of our in-class experimentsType “exit” to exit this shellAfter two hours, your interactive shell will be killed
06/14
cja 2014 27
Navigating the filesystem
06/14
cja 2014 28
Linux Filesystem Concepts
Files are stored in a directoryDirectories may contain other directories as well as files
A hierarchy of these directories is called a directory treeA directory tree (a connected graph with no cycles) has a single topmost root directory
A directory tree rooted at the system root directory “/” is called a filesystem
06/14
cja 2014 29 06/14http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html
cja 2014 30
Linux Filesystem ConceptsA file is accessed using its path name
Absolute path name/dir1/dir2/…/dirn/filename/usr/X11R6/bin
Relative path namecurrent-working-directory/filenamebin
Every shell maintains a notion of a current working directoryInitialized at login to your home directoryChanged via cd command
Two special directories. refers to the current directory.. refers to the current directory’s parent directory
06/14
cja 2014 31
Basic commands
06/14
cja 2014 32
PrerequisitesSome fundamental commands:~$ file file # what kind of file is file?
~$ cat file # display contents of text file
~$ less file # paginate text file
~$ man command # get info about command
Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC)
06/14
cja 2014 33
Lab 2Task: navigate the file systemCommands:~$ cd # make your home directory the current working directory
~$ pwd # print working directory
~$ mkdir foo # create directory foo
~$ cd foo # cd to the foo directory
~$ mkdir bar # create directory bar
~$ cd .. # cd back to the foo directory
~$ tree # display foo’s directory tree
06/14
cja 2014 34
Navigating the filesystem
Some fundamental commands:~$ pwd # print working directory
~$ cd dir # make dir the current working directory
~$ cd # cd to your home dir
~$ cd ~cja # cd to cja’s home dir
~$ mkdir dir # create directory dir
~$ rmdir dir # remove (empty) directory dir
~$ tree # display dir tree
06/14
cja 2014 35
Listing info on filesls – list information about files~$ ls # list contents of cur dir
~$ ls dir # list contents of dir
~$ ls –l # list details of files in cur dir
~$ ls –t # list newest files first
~$ ls –R dir # list all files in tree dir
~$ ls –lt dir # options can be combined
06/14
cja 2014 36
Working with filesThese commands manipulate files~$ mv foo bar # rename file foo to bar
~$ cp foo bar # copy file foo to bar
~$ cp –r dir1 dir2 # copy dir tree dir1 to dir2
~$ cp f1 f2 dir # copy f1 and f2 to directory dir
~$ mkdir dir # create empty directory dir
~$ rmdir dir # remove empty directory dir
~$ rm file # remove file file
~$ rm –r dir # remove directory tree dir
06/14
cja 2014 37
Lab 3Exercise:Create a directory named tutorial in your home directory. In that directory, create a directory named sample and a directory named test . Create a file named msg in directory test that contains a copy of the file /etc/motd .
06/14
cja 2014 38
Compressing and archiving
These commands compress and archive files~$ gzip foo # compress foo to foo.gz
~$ gunzip foo # uncompress foo.gz to foo
~$ tar cf foo.tar bar# archive subtree bar in file foo.tar
~$ tar xf foo.tar # restore archive from file foo.tar
~$ tar zcf foo.tgz bar # archive and compress
~$ tar jcf foo.tjz bar # archive and compress better
06/14
cja 2014 39
WildcardsAll Linux commands take wildcarded argumentsWildcards:
? Matches a single character* Matches zero or more characters[chars] Matches any of the chars[c1-c2] Matches chars ‘c1’ through ‘c2’[^chars] Matches any but the chars
~$ ls foo.? # match files named foo.x, where x is any character
~$ echo *.[cs] # echo files that end in .c or .s
~$ mv [o-z]* save # move files starting with o through z to directory save
~$ echo [^A-Z]? # ???
06/14
cja 2014 40
Shell redirection & pipelining
06/14
cja 2014 41
Shell redirectionA Linux command can have its inputs and outputs redirected~$ ls >foo # put list of files in current
directory into file foo
~$ ls >>bar # add list of files in current directory to end of file foo
~$ sort <bar # sort lines from file bar
~$ sort <<EOF # sort lines entered at keyboardwhiskeybravotangoEOF
~$ wc -l </etc/motd >~/mycounts# count number of lines from file /etc/motd and put result in file mycounts in my home directory
06/14
cja 2014 42
More Linux commands
More useful Linux tool commands~$ grep string # show lines of input containing
string
~$ tail # show last few lines of input
~$ head # show first few lines of input
~$ sort # sort the input
06/14
cja 2014 43
Shell pipeliningA Linux command can have its output connected to the input of another Linux command~$ ls | wc –l # count files in current
directory
~$ last | grep reboot # when did we reboot?
Exercise:How many people are running MATLAB on your compute node?
06/14
cja 2014 44
Editing text files
06/14
cja 2014 45
Editing text files
Simple editornano or pico
"What you see is what you get” editorSimple to learn if you want to get started quickly
Supported editorsvi or vimemacs
Powerful but more complexIf you have time and inclination to become proficient, spend time here
06/14
cja 2014 46
Text filesWatch out for source code or data files written on Windows systems
Use these tools to analyze and convert source files to Linux formatfiledos2unix
06/14
cja 2014 47
Permissions
06/14
cja 2014 48
File PermissionsThree permission bits, aka mode bits
Files: Read, Write, ExecuteDirectories: List, Modify, Search
Three user classesUser (File Owner), File Group, Other
man chmod
06/14
cja 2014 49
File Permissions, examples
-rwxr-xr-x cja lsaitfile read, write, and execute rights for the owner, read and execute for others
-rwxr-x--x cja lsaitsame permissions as above, except only the owner can read the file, while all can executechmod u=rwx,g=rx,o=x file
drwxr-x--x cja lsaitlist, modify, and search for the owner, list and search for group, and execute only for others
06/14
cja 2014 50
Processes
06/14
cja 2014 51
Processes
On Linux, every program runs in a process
You can examine these processes
man pspsps axtop
06/14
cja 2014 52
Processes
You can signal a running processTo stop it, or "kill" it
man kill
06/14
cja 2014 53
Any Questions?Charles J. AntonelliLSAIT Advocacy and Research [email protected]://www.umich.edu/~cja734 763 0607Please evaluate our course:
http://umichlsa.qualtrics.com/SE/?SID=SV_3pzjrMdjGXtUDAh
06/14
cja 2014 54
References1. http://en.wikipedia.org/wiki/History_of_Linux2. http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html3. William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No
Starch Press, January 2012. Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf.
06/14
cja 2014 55
ExtraTask: compile and execute simple programs on the Flux compute node
Copy sample code to your login directory:cdcp ~brockp/cac-intro-code.tar.gz .tar -xvzf cac-intro-code.tar.gzcd ./cac-intro-code
Examine, compile & execute helloworld.f90:ifort -O3 -ipo -no-prec-div -xHost -o f90hello helloworld.f90./f90hello
Examine, compile & execute helloworld.c:icc -O3 -ipo -no-prec-div -xHost -o chello helloworld.c./chello
Examine, compile & execute MPI parallel code:mpicc -O3 -ipo -no-prec-div -xHost -o c_ex01 c_ex01.cmpirun -np 2 ./c_ex01
06/14