The Linux Talk

Linux – Why should I care As of the Fall of 2010, the ME department has over 700

total processor cores running the Linux OS. The larger processor and memory based systems tend to

use Linux rather than windows. In the ‘bang for the buck’ arena, servers that do not

require terminals are far more economical for computational needs. A 24 core/48gb Linux server costs about the same as three dual core windows desktops.

ECN provides licenses for certain advanced software and compilers only on the Linux environment.

Linux offers a true multi-user OS. You can start jobs and let them run for months – literally. We have machines that have been up for over 300 days straight without a reboot or power cycle.

Linux vs. Windows

Differences: Linux is a multi-user system with a multi-

user kernel Windows is a single user system with a

single user kernel

Commonalities: Its all just a framework to do something Many applications are cross-platform – the

OS merely provides the environment to run

Linux vs. Windows Advantages????? Linux:

Its free and its widely available Its core is suitable for servers, workstations and

appliance applications Linux can be run as a larger petabyte capacity

system or configured to run off a 1.44mb floppy disk with no HD locally.

Windows: Very common, pervasive OS. Its pretty much what

most computers run. Very large assortment of software available

Linux vs. Windows

Disadvantages????? Linux:

Its complex. Simply put – the more capable you are, the more complex it becomes.

Many features are command line driven. Windows:

At its core – its really a single user operating system. (Fast user switching does not fix this)

Most features are GUI driven.

Linux vs. BSD or MAC or Solaris Linux is a version of UNIX, much like BSD is a version of UNIX. The

defining characteristic of Linux is its kernel. The Linux kernel is used in different flavors (versions) of Linux such as Fedora, Red Hat, SUSE, Debian, Ubuntu etc. OpenBSD, FreeBSD, MacOS do not use the Linux kernel and therefore are not Linux.

One of the defining factors is the kernel (hardware is the other). The kernel is the ‘core’ or ‘guts’ of the operating system. It provides the gateway between the software and the hardware. Common ‘flavors’ share the same kernel and in general, can share applications without recompiling (may require LOTS of work with library locations though!).

Mac uses a BSD based kernel. Mac also uses the Aqua display manager rather than the traditional X display manager common to other UNIX variants. What this means is MAC is a UNIX OS but it is not Linux and compiled apps for Linux likely will not run on MacOS.

Solaris is another BSD variant but also requires Sparc hardware. (non-PC). There is a large collection of UNIX variants on non-pc hardware (IRIX, AIX, Tru64, Solaris, HP-UX, VMS, VAX etc)

The ECN Linux Environment Relationship of disk space/home directory to machine itself. CPU’s (processors) are tied to machine, jobs run on CPU’s so jobs are tied to

machines. You login to a machine to run a job on a machine Disk Space transcends the machine itself. Files reside on disks and disk space is

NOT tied to a machine so that single file can be available on multiple machines at the same time. This is accomplished using NFS (network file system) server app.

Machine 1CPU(s)

Disk Space

Machine 2CPU(s)

Disk Space

Machine 3CPU(s)

The ECN Linux Environment The UNIX/Linux file systems form a ‘Tree’ where disk space on one

machine is available on all machines. Each File has an ‘address’ to find it. That address is its absolute

UNIX path.home camp

coep

project

gadget

robusta

tribe

a

a

a

a

b

a

a

/home/camp/a

/home/project/a

/home/gadget/a

/home/robusta/a

/home/robusta/b

/home/coep/a

/home/tribe/a

The ECN Linux Environment Backups:

ECN offers backup services for a fee▪ Backups are done each night and restores are available

through the ECN webpage. Some machines are backed up each night some

are not. For specifics – ask your advisor or the e-shop. Most research machines are NOT backed up. Most departmental file servers are.▪ Costs are License ($200 or $600) + $2/gb of capacity.

For instance, coep server with a 750gb hd costs $600 for the license and $1500 for data, total cost $2100

The ECN Linux EnvironmentCLUSTER MACHINES NON-CLUSTER

MACHINESrobusta

robusta01

robusta03robusta02

robusta04

robusta06robusta05

What’s the difference??? - mostly just how they are used!!!

tribecoep

navyseanm

pande

sharifcater

robusta07 apfel

The ECN Linux EnvironmentCLUSTERS

Tend to have a common home directory on the ‘head node’ that each ‘node’ points to.

Tends to have dedicated network segments

*may* have specialized high speed interconnect (robusta, steele, coates)

NON-CLUSTER Each machine has its

own home directory or points to the departmental file server

Tends to share their network segment with other machines

Parallel jobs *are* run on non-cluster machines

There is a LOT of overlap in usage patterns of machines

Getting Connected There are a few main ways to ‘connect’ to

a Linux/UNIX machine Console login (sitting in front of the machine) Remote text only connection (SSH) Remote Graphical connection (X – always

tunneled through SSL/SSH) Connecting to windows machine via Linux?

Use the ‘rdesktop’ command from the Linux command line▪ IE: rdesktop 128.46.184.220

Getting Connected Text only (SSH) – command line

This is the ‘telnet’ type interface but encrypted via SSL. In its base mode – its text only which makes it *very*

quick over fast connections and quite usable even over dialup connections.

SSH provides the base to ‘tunnel’ graphical applications when paired with the proper client.

SFTP is a secure FTP client variation based on the SSH protocol

Applications include: Cygwin, SecureCRT, SSH Secure Shell and SecureFX for Windows. Unix systems usually have the ssh command built in as well.

Getting Connected Graphical – remote connections

Linux is multi-user by default so a single Linux machine can support many simultaneous graphical sessions

Remote connections REQUIRE the connection to be ‘tunneled’ via SSH. From cygwin on a PC, this can be accomplished using the ‘ssh –X’ command.

Graphical connections can be VERY network intensive so beware of attempting complex rendering over slow network links.

The graphical ‘command line’ window is known as an xterm (X terminal).

Requires a local ‘X’ server to be running. For PC’s, an application such as Cygwin or Xceed is required. The ‘server’ application needs to be on the machine you are sitting at. The ‘client’ application is on the remote system.

Linux Shells and Environment The shell is the command

interpreter. Unlike windows, Linux offers many different options with different feature sets. tcsh, csh (C shell and C shell variants) sh, ash, dash, bash, zsh (Bourne Shell

and Variants) ksh (Korn shell) Plus a host of restricted shells,

customized feature set shells and other ‘exotic’ shells.

Linux Shells and Environment

By default, users start with the tcsh in ME (this is because this is the shell I use and its easier on me to get people started)

Customization files for tcsh .login, .cshrc, .tcshrc, .logout

Customization files for the bash shell .profile, .bash_profile, .bash_login, .bash_logout, .bashrc

Customization files for the zsh shell .zprofile, .zlogin, .zshrc

Customization files for the ksh shell .profile

Customization files for the sh shell .profile

Linux Shells and Environment Customizations

All user controlled customization files should be in the root of your home directory

Can set the path variable which defines where to search for programs/applications

Can set aliasing for commonly used programs to simpler commands

Can configuring your command prompt with history or different formats

Can execute commands on login or logout

Linux Shells – Environment The Linux shell has a set of predefined

variables. These variables are known as your environment variables. These can be setup on the command line or in one of the customization files for the shell. If you custom build software, you may need to

modify your environment If you use certain software packages, you may

need to setup variables in your environment You may want to use this for customizing your

‘experience’ while working in Linux

Linux Shells – Permissions Linux has three levels of

permissions, the user level, group level and world level. Each level offers three options – read (y/n), write (y/n) and

execute (y/n) It is legal and permissible to be allowed to write to a file but not

read it. For directories, traversal rights are controlled by the execute bit Permission set using the chmod command.

▪ chmod g+r <filename>▪ chmod –R 775 <filename>

Permission blocks

World

User

Group

Flag

d rwx r-x r-x

drwxr-xr-x

Linux Shells and Scripting Linux shells are applications

themselves. They can be ‘stacked’ such that you may use tcsh as the default but can invoke a bash shell as needed on top of the tcsh login shell.

Scripts work in this regard. They create their own non-login shells in which to run. You can invoke scripts within your current shell or to run in their own environment space.

Linux Shells and Scripting The basics

▪ (books and classes are taught on this subject) A shell script, to run it its own environment space must

have a ‘sha-bang’ line as the first characters/line of the file.▪ #!/bin/sh▪ #!/usr/local/bin/tcsh

A Shell script must be ‘executable’ - IE the ‘x’ bit set Each shell has varying capabilities of loops, variables, math

functionality and external communications available. The ‘sh’ shell is the most common with the ‘ksh’ being the most common for programming.

Python, Perl, PHP etc are interpreters, not shells. They can do scripting and ‘application’ functions but lack the user level interactivity that normally defines a ‘shell’

Linux Shells, Basic Commands A simple listing of the fundamental

commands that are useful to know

ssh

cd catpwd

chmod man vi grep

rm rmdir

pico tail

head

ls

sftp logout more less

nohupps kill top uptime who

Linux Shells, Basic Commands Examples:

Listing all files in the current directory, with permission bits and hidden files• ls -al

Showing all of the processes on a machine for the user loganm• ps –ef | grep loganm

killing a task (PID from above command)• kill -9 <PID> IE: kill -9 23999

Removing a directory and all files/directories it contains• rm –rf <directory name> IE: rm –rf test_program1

Making a script executable• chmod u+x <scriptname> IE: chmod u+x testprog.sh

starting a process independent of the terminal session• nohup <script> IE: nohup important_job.ksh

killing a task (PID from above command)• kill -9 <PID> IE: kill -9 23999

Finding a file recursively• find . -name "mysql.h" -print

Linux Shells + Scripts

The shell environment also lends itself to interpreted scripts. These are languages which offer much greater flexibility and capability over simple shell scripts but lack the complexity of compiled languages. Common languages supported include (but not limited to)

PERL Python PHP Ruby Java

Linux Shells and Custom Code We also support the compiled

languages. These are the most versatile and the most complicated to use. There is full support for mathematical libraries (BLAS, LAPACK), socket programming, database connectivity etc. These use compilers to create the executable and we support the following suites.GNU

SuiteIntel Suite

Portland Group

Linux Software – Where to find it There are several common locations for

software to be found. Many are in the default ‘path’ of the machine. /usr/bin /bin /usr/local/bin /usr/opt/bin

Packages are installed in either the Red Hat default (core Linux applications) or in /package (nfs shared disk)

Linux Software – Building your Own One of the great features of Linux is the

‘sandbox’ capability of applications. Users can build and run fully functional versions of applications within their own user space. Each application is dependent only on itself.

If you build your own packages, be sure to set the install directory (prefix) with the configure script. The defaults will *never* work.

The basic process to build and application is belowInstall

prerequisite

software

Run the ‘configure’ script with proper local install options

Run the ‘make’

command

Run the ‘make install’ command

to install the software

Linux Graphical Display Managers The X display server’s built in display manager is

very basic. To enhance usability, several new ‘Graphical Display Managers’ have been developed. KDE Gnome CDE Many others

These GDM’s offer user level customizations with significant options. I can go through some examples during the Q&A portion for adding applications and managing menus for Gnome.

Parallel Computing with Linux Linux offers several means for utilizing multiple CPU cores

to accomplish your tasks MPI based code Parallel aware applications Threads

It should be noted – the efficiency of multiple cpu usage is related to your code/job. Some jobs parallelize easily, others do not. It is up to you, the designer of the job, to determine what is the best way to run your job

Parallel computing is not free. Parallel aware applications require more licenses for parallel jobs. The speed increase is not linear and in some cases, can be speed decreases.

MPICH2 and Linux MPI (message passing interface) is the most

common way to do parallel computing with C and Fortran based codes.

Can be used on the same machine, across Ethernet with multiple machines or through Infiniband

To use MPI – you must compile your code with the ‘wrapped’ MPI aware compiler. All of the major compilers have ‘wrapped’ versions available.

MPICH2 does require configuration of your user environment and the use of a ‘daemon’ application on each node. (MPD and MPD rings)

Multi-CPU Abaqus + Fluent Abaqus and Fluent both support native

and easy parallel computing support Abaqus offers two means for multi-cpu usage▪ cpus=n command line option ▪ mpi implementation using HP-MPI (not

implemented on ECN) Fluent offers a parallel solver (see the Fluent

docs for details) Be aware – parallel processing requires

more licenses from a limited pool!

Paralell Toolbox for Matlab Matlab is fundamentally a uni-processor

application. Recently – Matlab added some rudimentary

parallel computing functionality by using the parallel computing toolbox and ‘worker’ instances of Matlab Parallel ‘for loops’ Batch command parallel execution Distributed data for very large arrays

Read the toolbox documentation!

Other Parallel Variants Threads - think of the fork command but with

communication. Different languages provide different options

Named pipes – much like threads but at a lower level. Again, implementation is VERY language dependent.

Socket level programming – fundamentally, this is an implementation strategy using standard network libraries. MPI is built on this framework.

SCSI backplane or fabric backplane – this is the level Infiniband and Myrinet are implemented on. There can be some advantages to gain in latency and bandwidth. This is also the structure GPU programming falls. (CUDA chips)

Resources & Machines We have over 700 processor cores running in ME 242A and over 750 cores in servers in ME

department as a whole. On top of this – we have and support another 30+ Linux workstations, many of which are multi-core machines.

Publicly Available Departmental Linux Machines ME – (5) Dual core workstations (steam, wind, water, air, cog), (1) 24-core server (horsepower) Herrick – (2) single core workstations (stave, manon), (4) 8-core servers (herrick, cohen, bernhard, fontaine) ECN – (1) 16-core server (riptide) <shared by the college of engineering>

Research Groups (servers only) Sadeghi – 164 cores in 12 servers (coep, tribe, pande, sharif, cater, apfel, navy, seanm, cat, schaeffler,

ashtekar, evansville) Frankel – 124 cores in 17 nodes – Supremo + Robusta clusters + coffeeexpress. (robusta has Infiniband) Wassgren – 76 cores in 14 nodes - Camp cluster Key – 72 cores in 9 servers (Ubuntu non-networked cluster - Zucrow) Shin – 68 cores in 6 servers (femtosim, picosim, lampsim, mansim, microsim, ultrasim) Lucht – 56 cores in 7 servers (densitymatrix – densitymatrix7) Ramani – 48 cores in 2 servers (shape, kernel) Xu – 48 cores in 2 servers (bncws1, bncws2) Mongea – 24 cores in 1 server (tfm) Siegmund – 16 cores in 2 servers (fracture, asterix) Ruan – 16 cores in 4 servers – nanoenergy cluster Martini – 8 cores in 1 server (vader) Subbarayan – 8 cores in 1 server (magenta) Son – 4 cores in 1 server (gadget) Fisher – 4 cores in server (edwards) Li – 2 cores in 1 server (noise)

Q&A + Examples