Cluster Computing: Updraft - University of Utah › files › 2015 › 01 › updraft.pdf · Cluster Computing: Updraft Charles Reid Scientiﬁc Computing Summer Workshop June 29,

Cluster Computing: UpdraftCharles ReidScientific Computing Summer WorkshopJune 29, 2010

Wednesday, June 30, 2010


Updraft Cluster: Hardware

• 256 Dual Quad-Core Nodes

• 2048 Cores

• 2.8 GHz Intel Xeon Processors

• 16 GB memory per node

• Qlogic Infiniband network interconnect

• Ethernet network interconnect


• 2048 Cores








• 2048 Cores






• 2048 Cores






Node Networking

• Networking “flavors” for nodes:

• Ethernet

• Infiniband


Node Networking

• Networking “flavors” for nodes:

• Ethernet

• Infiniband


Head Nodes

• Updraft 1 (updraft1.chpc.utah.edu)

• Updraft 2 (updraft2.chpc.utah.edu)

• NOT FOR COMPUTATION!!!

• Illustrative Example




• 2048 Cores



• Qlogic Infiniband interconnect

• Gigabit ethernet interconnect


Simple Parallel Example

• What actually happens when you run programs on multiple cores?


Golden Rule:

• The one with the gold makes the rules



Funding for Updraft:

CSAFEICSE

Others


Updraft Policies

• ICSE and CSAFE users: running big jobs

• Updraft designed for big jobs

• High throughput vs. High capacity

• Preemption: some jobs can preempt other jobs

• QOS: your “quality of service” determines whether you can pre-empt other jobs


Updraft Policies: Allocations

• Allocation = how much time you get on Updraft

• Every CPU hour run decreases your allocation

• Allocations given for each quarterJan-Mar, Apr-Jun, Jul-Sep, Sep-Dec

• Allocations distributed/grouped by P.I. (primary investigators), not on individual basis

• CHPC website: current allocation statushttp://www.chpc.utah.edu


http://www.chpc.utah.edu


Updraft Policies: Preemptors

QOS Priority

bigrun+++

(preemptor)NOT FOR SMALL JOBS!!!

Jobs > 512 nodes

uintah ++(preemptor)

Must be in Uintah group

general++

(preemptor) Anyone


Updraft Policies: Preemptees

QOS Priority

preemptable+

(preemptable)Charges allocation

at 1/4 the rate

freecycle0

(preemptable)Doesn’t use/require

any allocation


Updraft Policies:DATs

• Dedicated Access Time

• 48 hours to access 100% of Updraft

• 1st week of the month - Uintah DAT

• 3rd week of the month - Uintah DAT

• 4th week of the month - General DAT

• How to find out when DATs are?



Updraft Queue System

• Queue system: schedules and manages jobs and resources

• Other systems: Moab, Maui, etc.

• PBS scripts - see CHPC website for examples

• Submitting jobs - qsub

• Deleting jobs - qdel

• Checking on jobs - checkjob

• Showing entire queue - showq / qstat


• When you run a job, where do you put stuff?

• NFS (Network file system)

• Uintah disk space

• Scratch & temp disk space

Updraft Hard Disks


• Slowest... 100 - 200 MB/s

• NOT for dumping simulation output!!!

• Home directories:/uufs/chpc.utah.edu/common/home

• CRSim group space:/uufs/chpc.utah.edu/common/home/crsim_grp

• ICSE group space:/uufs/chpc.utah.edu/common/home/icse_grp

Updraft Hard Disks: NFS


• Faster access... for executables and code

• NOT for simulation output!!!

• Limited disk space for code development/execution

• Only accessible from updraft2.chpc.utah.edu head node

• Location:/uufs/updraft.arches/common/uintah/homebrew

Updraft Hard Disks: Uintah Disk Space


• This IS for simulation output!!!

• Disk speeds are faster: 200-800 MB/s

• WARNING: Cleaned monthly/at random!

• Uintah scratch space:/scratch/uintah

• General scratch space:/scratch/general

• Node scratch space (very small):/tmp

Updraft Hard Disks: Scratch/Temp Disk Space


• Disk usage summaries are emailed weekly

• Email [email protected] to be added to email list for disk usage summaries

• disk usage (for files/folders): du

• disk free space (for file systems): df

Updraft Hard Disks: Getting Disk Info


mailto:[email protected]

mailto:[email protected]

• Compilers

• GCC compiler suite (gcc)

• Intel compiler suite (icc)

• Pathscale compilers (pathcc)

• Portland group compilers (pgcc)

Updraft Software


• MPI

• MPICH

• MVAPICH/MVAPICH2

• OpenMPI

• Qlogic

Updraft Software


• Linear algebra packages

• ATLAS

• BLAS

• LAPACK/ScaLAPACK

• IMKL (Intel Math Kernel Library)

• Hypre

• PETSc

Updraft Software


• Full list of software: http://www.chpc.utah.edu

• Presentations (many cover software mentioned here): http://www.chpc.utah.edu

• Locations for stuff:/uufs/chpc.utah.edu/sys/pkg/uufs/updraft.arches/sys/pkg

• Alternatively: build your own stuff $HOME/pkg

Updraft Software

Subject of next workshop!







$ system_profiler

$ cat /proc/cpuinfo

$ top

$ who

$ finger

$ showres

$ qsub

$ qdel

$ checkjob

$ qstat

$ showq

$ du -h

$ df -h


★ http://www.chpc.utah.edu

★Cluster Hardware:http://chpc.utah.edu/docs/manuals/hardware/

★Updraft User’s Guide:http://chpc.utah.edu/docs/manuals/user_guides/updraft

★Queue system:http://chpc.utah.edu/docs/manuals/user_guides/updraft/#batch

★Available Software:http://chpc.utah.edu/docs/manuals/software/

★Presentations:http://chpc.utah.edu/docs/presentations/

★CHPC Wiki:https://wiki.chpc.utah.edu/




http://chpc.utah.edu/docs/manuals/user_guides/updraft

http://chpc.utah.edu/docs/manuals/user_guides/updraft

http://chpc.utah.edu/docs/manuals/software/

http://chpc.utah.edu/docs/manuals/software/

https://wiki.chpc.utah.edu

https://wiki.chpc.utah.edu

Documents

Cluster Computing: Updraft - University of Utah › files › 2015 › 01 › updraft.pdf · Cluster Computing: Updraft Charles Reid Scientiﬁc Computing Summer Workshop June 29,