Sheffield HPC Documentation Release

Sheffield HPC DocumentationRelease

Feb 12, 2018

Contents

1 Research Computing Team 3

2 Research Software Engineering Team 5

i

ii

Sheffield HPC Documentation, Release

This is the documentation for The University of Sheffield’s High Performance Computing (HPC) clusters, ShARC andIceberg. Run by CiCS’ Research Computing Group with additional support from the Research Software Engineeringteam in Computer Science, they support the computational needs of hundreds of researchers across all departments.

Contents 1


2 Contents

CHAPTER 1

Research Computing Team

The research computing team are the team responsible for the HPC services, as well as all other aspects of researchcomputing. If you require support with HPC, training or software for your workstations, the research computing teamwould be happy to help. Take a look at the Research Computing website or email [email protected].

3

https://www.shef.ac.uk/cics/research

mailto:[email protected]


4 Chapter 1. Research Computing Team

CHAPTER 2

Research Software Engineering Team

The Sheffield Research Software Engineering Team is an academically led group in the Department of ComputerScience that collaborates closely with CiCS. They can assist with code optimisation, training and all aspects of HighPerformance Computing including GPU computing along with local, national, regional and cloud computing services.Take a look at the Research Software Engineering website or email [email protected]

Using the HPC Systems

If you have not used a High Performance Computing (HPC) cluster, the Linux operating system or even a commandline before this is the place to start. This guide will get you set up using the University’s clusters (ShARC and Iceberg)fairly quickly.

Getting an Account

Before you can start the clusters you need to register for an account. An account gives you access to both clusters(ShARC and Iceberg). Accounts are availible for staff by emailing [email protected].

The following categories of students can also have an account with the permission of their supervisors:

• Research Postgraduates

• Taught Postgraduates - project work

• Undergraduates 3rd & 4th year - project work

A student’s supervisor can request an account for the student by emailing [email protected].

Note: Once you have obtained your username, you need to initialize your password to be the same as your normalpassword by using the CICS synchronize passwords system.

5

https://www.sheffield.ac.uk/dcs

https://www.sheffield.ac.uk/dcs

http://rse.shef.ac.uk/




https://www.shef.ac.uk/cics/password


Connecting to a cluster using SSH

The most versatile way to run commands and submit jobs on one of the clusters is to use a mechanism called SSH,which is a common way of remotely logging in to computers running the Linux operating system.

To connect to another machine using SSH you need to have a SSH client program installed on your machine. macOSand Linux come with a command-line (text-only) SSH client pre-installed. On Windows there are various graphicalSSH clients you can use, including MobaXTerm.

Note: You can connect to Iceberg using SSH (or the related protocols SCP and SFTP) from anywhere (on campusand off campus with/without a VPN connection) but if you are using ShARC and you are off-campus then you needto set up a VPN connection first (or establish one SSH connection to Iceberg then a second from Iceberg to ShARC).

SSH client software on Windows

Download and install the Installer edition of mobaxterm.

After starting MobaXterm you should see something like this:

Click Start local terminal and if you see something like the following then please continue to Establishing a SSHconnection.

6 Chapter 2. Research Software Engineering Team

https://en.wikipedia.org/wiki/Secure_Shell

https://www.sheffield.ac.uk/cics/vpn


https://mobaxterm.mobatek.net/download-home-edition.html


Running commands from a terminal (from the command-line) may initially be unfamiliar to Windows users but thisis the recommended approach for running commands on ShARC and Iceberg as it is the idiomatic way of interfacingwith the Linux clusters.

SSH client software on Mac OS/X and Linux

Linux and macOS (OS X) both typically come with a command-line SSH client pre-installed.

If you are using macOS and want to be able to run graphical applications on the clusters then you need to install thelatest version of the XQuartz X Windows server.

Open a terminal (e.g. Gnome Terminal on Linux or Terminal on macOS) and then go to Establishing a SSH connection.

Establishing a SSH connection

Once you have a terminal open run the following command to log in to a cluster:

ssh -X $USER@$SGE_CLUSTER_NAME.shef.ac.uk

Here you need to:

• replace $USER with your CiCS username (e.g. te1st)

• replace $SGE_CLUSTER_NAME with sharc or iceberg.

Note: macOS users: if this fails then:

• Check that your XQuartz is up to date then try again or

• Try again with -Y instead of -X

This should give you a prompt resembling the one below:

[te1st@sharc-login1 ~]$

At this prompt type:

qsh

Like this:

[te1st@sharc-login2 ~]$ qshYour job 135355 ("INTERACTIVE") has been submittedwaiting for interactive job to be scheduled ....Your interactive job 135355 has been successfully scheduled.

Which will pop up another terminal window, which supports graphical applications.

Note: When you login to a cluster you reach one of two login nodes. You should not run applications on the loginnodes. Running qsh gives you an interactive terminal on one of the many worker nodes in the cluster.

If you only need terminal-based (command-line only) applications you can run the qrsh command, which will giveyou a shell on a worker node, but without graphical application (X server) support.

2.1. Using the HPC Systems 7

https://www.xquartz.org/



What Next?

Now you have connected to a cluster, you can look at how to submit jobs with Starting interactive jobs and submittingbatch jobs or look at the software installed on ShARC and Iceberg

Filestores

Every user on ShARC/Iceberg has access to five different types of filestore:

• /home/yourusername

• /data/yourusername

• /fastdata

• /shared/volumename

• /scratch

They differ in terms of:

• the amount of space available;

• whether they are available from multiple nodes;

• whether they are shared between clusters;

• whether the underlying storage system is performant if reading/writing large files;

• whether the underlying storage system is performant if reading/writing small files;

• frequency of backup and the time that the data can be left there.

Here are the current details of filestore available to each user.

/home directory

All users have a home directory in the location /home/yourusername.

The filestore quota is 10 GB per user.

This area is shared between ShARC and Iceberg and is accessible to all worker and login nodes.

Backup policy: /home has backup snapshots taken every 4 hours and we keep the 10 most recent. /home also hasdaily snapshots taken each night, and we keep 28 days worth, mirrored onto a separate storage system.

The filesystem is NFS.

/data directory

Every user has access to a much larger data-storage area provided at the location /data/yourusername.

The quota for this area is 100 GB per user.

This area is shared between ShARC and Iceberg and is accessible to all worker and login nodes.

Backup policy: /data has snapshots taken every 4 hours and we keep the 10 most recent. /data also has dailysnapshots taken each night, and we keep 7 days worth, but this is not mirrored.

The filesystem is NFS.

Note: the directory /data/yourusername is made available to you (mounted) on demand: if you list thecontents of /data after first logging on then this subdirectory might not be shown. However, if you list the contents



of /data/yourusername itself or change into that directory then its contents will appear. Later on if you list thecontents of /data again you may find that /data/yourusername has disappeared again, as it is automaticallyunmounted following a period of inactivity.

/fastdata directory

All users also have access to a large fast-access data storage area under /fastdata. This is not shared betweenShARC and Iceberg.

In order to avoid interference from other users’ files it is vitally important that you store your files in a directorycreated and named the same as your username. e.g.

mkdir /fastdata/yourusername

By default the directory you create will have world-read access. If you want to restrict read access to just your accountthen run

chmod 700 /fastdata/yourusername

after creating the directory. A more sophisticated sharing scheme would have private and public directories

mkdir /fastdata/yourusernamemkdir /fastdata/yourusername/publicmkdir /fastdata/yourusername/private

chmod 755 /fastdata/yourusernamechmod 755 /fastdata/yourusername/publicchmod 700 /fastdata/yourusername/private

The fastdata area provides 669 TeraBytes of storage on SHARC and 260 Terabytes of storage on Iceberg. It takes ad-vantage of the internal high-speed network infrastructure (OmniPath interconnects for ShARC; Infiniband for Iceberg)for fast access to data.

/fastdata is optimised for large file operations. it is faster than /home/, /data and /shared (see below)when dealing with larger files but does not handle lots of small files very well: it is less efficient than /scratch(see below) when dealing with smaller files. An example of how slow it can be for large numbers of small files isdetailed at http://www.walkingrandomly.com/?p=6167

Iceberg users: although /fastdata is available on all the worker nodes, you need to run your jobs on the newernodes with Intel CPUs for best performance (by specifying -l arch=intel* in your job submission script).

Warning: There are no quota controls on the /fastdata area but older files on /fastdata are automaticallydeleted: a report of files older than 60 days is regularly generated, the owners of these files are then notified byemail then a week after the email(s) are sent the identified files are deleted.

We reserve the right to change this policy without warning in order to ensure efficient running of the service.

It is important to therefore not use /fastdata for long-term storage and copy important data on /fastdata tobacked-up areas such as /home, /data or /shared.

You can use the lfs command to find out which files under /fastdata are older than a certain number of days andhence approaching the time of deletion. For example, to find files 50 or more days old

lfs find -ctime +50 /fastdata/yourusername


http://www.walkingrandomly.com/?p=6167


Backup policy: /fastdata is not backed up.

/fastdata uses the Lustre filesystem. This does not support POSIX locking which can cause issues for someapplications (e.g. programs that create/use SQLite databases).

/shared directories

CiCS now provide 10 terabytes of shared storage for free per research group. After the storage has been re-quested/purchased by a group’s PI and then provisioned by CiCS it can be accessed by name

• as a Windows-style (SMB) file share on machines other than ShARC/Iceberg using \\uosfstore.shef.ac.uk\shared\;

• as a subdirectory of /shared on ShARC/Iceberg (you need to explicitly request HPC access when you orderstorage from CiCS).

Note that this subdirectory will be mounted on demand on ShARC/Iceberg: it will not be visible if you simply listthe contents of the /shared directory but will be accessible if you cd (change directory) into it e.g. cd /shared/my_group_file_share1

Regarding permissions: behind the scenes, the file server that provides this shared storage manages permissions usingWindows-style ACLs (which can be set by area owners via the Research Storage management web interface. However,the filesystem is mounted on a Linux cluster using NFSv4 so the file server therefore requires a means for mappingWindows-style permissions to Linux ones. An effect of this is that the Linux mode bits as seen on ShARC/Iceberg arenot always to be believed for files under /shared: the output of ls -l somefile.sh may indicate that a fileis readable/writable/executable when the ACLs are what really determine access permissions. Most applications haverobust ways of checking for properties such as executability but some applications can cause problems when accessingfiles/directories on /shared by naievely checking permissions just using Linux mode bits:

• which: a directory under /shared may be on your path and you may be able to run a contained executablewithout prefixing it with a absolute/relative directory but which may fail to find that executable.

• Perl: scripts that check for executability of files on /shared using -x may fail unless Perl is explicitly told totest for file permissions in a more thorough way (see the mention of use filetest 'access' here).

• git: may complain that permissions have changed if a repository is simply moved to /shared/someplacefrom elsewhere on ShARC/Iceberg. As a workaround you can tell git to not to track Linux permissions for asingle repository using git config core.filemode false or for all repositories using git config--global core.filemode false.

The documentation for the /shared storage serivce includes information on:

• how access/permissions are managed

• how to create folders with associated permissions within /shared storage areas

/scratch: for reading/writing small files

For jobs that need to read/write lots of small files the most performant storage will be the temporary storage on eachnode (under the /scratch directory).

This is because with /home, /data, /fastdata and /shared each time a file is accessed the filesystem needs torequest ownership/permissions information from another server and for small files these overheads are proportionallyhigh. However, for /scratch such ownership/permissions metadata is available on the local machine, so it is fasterwhen dealing with small files.

The most obvious disadvantage to the /scratch node-local storage is that a given directory cannot relabily beaccessed between jobs as you cannot guarantee that your next job will run on the same node. Any data of value musttherefore be copied off /scratch (e.g. to /home or /data) before the end of your job.


https://en.wikipedia.org/wiki/Lustre_(file_system)

https://sheffield.ac.uk/cics/research-storage/using-research-storage

https://en.wikipedia.org/wiki/Access_control_list

https://sheffield.ac.uk/storage

https://en.wikipedia.org/wiki/Modes_(Unix)

http://linux.die.net/man/1/which

http://perldoc.perl.org/functions/-X.html

https://www.sheffield.ac.uk/cics/research-storage/access-rights

https://www.sheffield.ac.uk/cics/research-storage/create-folders


Where to store data within ‘‘/scratch‘‘: The scheduler automatically creates a per-job directory for you under /scratch. If you started your job using qrshx, qsh or qsub then the name of this directory is stored in the$TMPDIR environment variable e.g.

[te1st@sharc-login1 ~]$ qrshx[te1st@sharc-node003 ~]$ cd $TMPDIR[te1st@sharc-node003 667443.1.all.q]$ pwd/scratch/667443.1.all.q

The scheduler will then clean up (delete) $TMPDIR at the end of your job, ensuring that the space can be used byother users.

If using qrsh to start your job then the environment variable will unfortunately be undefined so you will need tomanually create a directory under /scratch (named using your username) and this will not be cleaned up when thejob ends.

Anything under the /scratch may be deleted periodically when the worker-node is idle. /scratch is not backedup. There are no quotas for /scratch storage.

/scratch uses the ext4 filesystem.

Determining your current filestore allocation

To find out your current storage quota usage for /home and /data:

quota

If you exceed your file storage allocation

As soon as the quota is exceeded your account becomes frozen. In order to avoid this situation it is strongly recom-mended that you:

• Use the quota command to check your usage regularly.

• Copy files that do not need to be backed up to the /fastdata/username area, or remove them fromShARC/Iceberg completely.

Recovering snapshots

We take regular back-ups of your /home and /data directories and it is possible to directly access a limited subsetof them.

There are 7 days worth of snapshots available in your /home and /data directories in a hidden directory called.snapshot. You need to explicitly cd into this directory to get at the files:

cd /home/YOURUSERNAME/.snapshot

The files are read-only. This allows you to attempt recover any files you might have accidentally deleted recently.

This does not apply for /fastdata for which we take no back-ups.

Transferring files

To transfer files to/from the clusters you can either:



• Use a program that supports one or both of the SCP and SFTP protocols to copy/move files to/from your ownmachine.

• Use a Research Storage fileshare as common storage directly accessible from your own machine and from theclusters.

A more comprehensive guide to accessing the clusters and transfering files is located at http://www.sheffield.ac.uk/cics/research/hpc/using/access.

Using interactive sessions / submitting batch jobs

DRMAA

The Distributed Resource Management Application API (DRMAA) is a specification for programatically submit-ting/controlling jobs to/on job scheduling software such as the Grid Engine software used on the ShARC and Icebergclusters.

Support for DRMAA is built in to Grid Engine and other scheduling software such as Torque, PBS, HTCondor andSLURM.

There are DRMAA bindings (i.e. wrappers) available for several programming languages.

Why is it useful?

Using DRMAA you may find it easier to automate the submission and control of jobs.

Also, you might want to use a Computational Pipeline manager (such as Ruffus) to run sets of related (possiblydependent) jobs on a cluster; these may use DRMAA beind the scenes to create and manage jobs.

Using DRMAA from Python

Install DRMAA bindings

The Python bindings for DRMAA are compatible with Python 2 and Python 3. The simplest way to install them is touse a conda environment e.g.

# Connect to ShARCssh sharc# Start an interactive sessionqrshx# Ensure conda is availablemodule load apps/python/conda# Create a conda environmentconda create -n drmaa-test-sharc python=3.6# Activate this environmentsource activate drmaa-test-sharc# Install the python bindings for DRMAApip install drmaa

Submitting a job

You can then submit a job using a script like this:


http://www.sheffield.ac.uk/cics/research-storage/

http://www.sheffield.ac.uk/cics/research/hpc/using/access


https://en.wikipedia.org/wiki/Language_binding

http://www.ruffus.org.uk/

http://drmaa-python.readthedocs.io/en/latest/


Listing 2.1: drmaa_submit_job.py

1 #!/usr/bin/env python2

3 from __future__ import print_function4 import drmaa5 import os6

7

8 def main():9 """

10 Create a DRMAA session then submit a job.11 Note, need file called myjob.sh in the current directory.12 """13 with drmaa.Session() as s:14 print('Creating a job template')15 jt = s.createJobTemplate()16 # The job is to run an executable in the current working directory17 jt.remoteCommand = os.path.join(os.getcwd(), 'myjob.sh')18 # Arguments to the remote command19 jt.args = ['alpha', 'beta']20 # Join the standard output and error logs21 jt.joinFiles = True22

23 job_id = s.runJob(jt)24 print('Job {} submitted'.format(job_id))25

26 print('Cleaning up')27 s.deleteJobTemplate(jt)28

29

30 if __name__ == '__main__':31 main()

where myjob.sh is:

#!/bin/bashecho "Hello from ${JOB_ID}. I received these arguments: $1, $2"

To actually submit the job:

$ # Make your job script executable$ chmod +x myjob.sh

$ python drmaa_submit_job.pyCreating a job templateJob 401022 submittedCleaning up

$ cat myjob.sh.o401022Hello from 401022. I received these arguments: alpha, beta

You can submit multiple jobs iteratively, reusing your DRMAA job template for efficiency.

If you want to submit a job array rather than a single job then you can call the runBulkJob method instead of runJob.


http://drmaa-python.readthedocs.io/en/latest/tutorials.html#running-a-job


Warning: Starting a single DRMAA session then attempting to submit jobs from different Python threads cur-rently does not work.

Email notifications

You can enable email notifications for jobs finishing or aborting by setting two attributes of your JobTemplateobject (jt in this example):

jt.blockEmail = Falsejt.email = ['[email protected]']

This is equivalent to having the following in a Grid Engine batch job submission script:

#$ -m ea#$ -M [email protected]

Other useful job template attributes

• Requesting arbitrary resources from Grid Engine such as real memory (rmem), parallel environments (e.g. -pempi), run time (h_rt), Grid Engine Projects e.g.

jt.nativeSpecification = '-l h_rt=00:30:00 -l rmem=2G -pe smp 2'

• Set the paths to the job standard output and error files:

jt.outputPath = os.path.join(os.getcwd(), 'output.log')jt.errorPath = os.path.join(os.getcwd(), 'error.log')

• Send standard output and error to the same file:

jt.joinFiles = True

• Name your job (equivalent to -N my_job in a Grid Engine batch job submission script):

jt.jobName = 'my_job'

• Set one or more environment variables in your job using a Python dictionary:

jt.jobEnvironment = {'SOMEKEY': 'somevalue', 'ANOTHERKEY', 'anothervalue'}

Warning: You cannot presently request one or more GPUs using DRMAA without explicitly using the gpuGrid Engine Project i.e. jobs created using a Job Template jt where jt.nativeSpecification = '-lgpu=1' will be rejected by the job scheduler but jobs with jt.nativeSpecification = '-P gpu -lgpu=1' will be accepted. This is due to a bug in the scheduler.

Waiting on jobs

Within a DRMAA Session you can wait indefinitely or forever for one, several or all submitted jobs to complete. Seethe documentation for more information including an example. Jobs are identified by (Grid Engine) job ID, so they donot need to have been submitted by DRMAA.


https://github.com/pygridtools/drmaa-python/issues/44

https://github.com/pygridtools/drmaa-python/issues/44

https://docs.python.org/3/tutorial/datastructures.html#dictionaries

https://arc.liv.ac.uk/trac/SGE/ticket/1342

http://drmaa-python.readthedocs.io/en/latest/tutorials.html#waiting-for-a-job


Controlling jobs

Within a DRMAA Session you can also terminate, suspend, resume, hold and release a job. See the documentationfor more information including an example. Again, jobs are identified by (Grid Engine) job ID, so they do not need tohave been submitted by DRMAA.

Checking the status of a job

Within a DRMAA Session you can check to see if any job is queuing, running, has completed successfully, has failedor is in some other state. See the documentation for more information including an example.

Futher information

See the documentation for the DRMAA Python bindings; you may find the enclosed reference information useful.Note that not all features of these bindings are described in this documentation!

Java bindings

Java bindings for DRMAA are not currently available on ShARC or Iceberg.

Administrator notes

Grid Engine’s implementation of the DRMAA shared library lives at $SGE_ROOT/lib/lx-amd64/libdrmaa.so.

It is discovered by bindings using the DRMAA_LIBRARY_PATH environment variable, which is set to the above forall users using a shell script in /etc/profile.d/.

qhost

qhost is a scheduler command that show’s the status of Sun Grid Engine hosts.

Documentation

Documentation is available on the system using the command:

man qhost

Examples

Get an overview of the nodes and CPUS on Iceberg

qhost

This shows every node in the cluster. Some of these nodes may be reserved/purchased for specific research groups andhence may not be available for general use.


http://drmaa-python.readthedocs.io/en/latest/tutorials.html#controlling-a-job

http://drmaa-python.readthedocs.io/en/latest/tutorials.html#getting-job-status

http://drmaa-python.readthedocs.io/en/latest/

http://drmaa-python.readthedocs.io/en/latest/drmaa.html


qrsh

qrsh is a scheduler command that requests an interactive session on a worker node. The resulting session will notsupport graphical applications. You will usually run this command from the head node.

Examples

Request an interactive session that provides the default amount of memory resources

qrsh

Request an interactive session that provides 10 Gigabytes of real memory

qrsh -l rmem=10G

qrshx

qrshx is a scheduler command that requests an interactive session on a worker node. The resulting session willsupport graphical applications. You will usually run this command from a login node.

Examples

Request an interactive X-Windows session that provides the default amount of memory resources and launch thegedit text editor:

qrshxgedit

Request an interactive X-Windows session that provides 10 Gigabytes of real memory and launch the latest version ofMATLAB:

qrshx -l rmem=10Gmodule load apps/matlabmatlab

Request an interactive X-Windows session that provides 10 Gigabytes of real memory and 4 CPU cores:

qrshx -l rmem=10G -pe openmp 4

Sysadmin notes

qrshx not a standard SGE command; it was specific to the University of Sheffield’s clusters.

On ShARC qrshx resides at /usr/local/scripts/qrshx and has this content.

On Iceberg it resides at /usr/local/bin/qrshx and is:

#!/bin/shexec qrsh -v DISPLAY -pty y "$@" bash


https://gist.github.com/willfurnass/10277756070c4f374e6149a281324841


qsh

qsh is a scheduler command that requests an interactive X-windows session on a worker node. The resulting terminalis not user-friendly and we recommend that you use our qrshx command instead.

Examples

Request an interactive X-Windows session that provides the default amount of memory resources

qsh

Request an interactive X-Windows session that provides 10 Gigabytes of real memory

qsh -l rmem=10G

Request an interactive X-Windows session that provides 10 Gigabytes of real memory and 4 CPU cores

qsh -l rmem=10G -pe openmp 4

qstat

qstat is a scheduler command that displays the status of the queues.

Examples

Display all jobs queued on the system

qstat

Display all jobs queued by the username foo1bar

qstat -u foo1bar

Display all jobs in the openmp parallel environment

stat -pe openmp

Display all jobs in the queue named foobar

qstat -q foobar.q

qsub

qsub is a scheduler command that submits a batch job to the system.

Examples

Submit a batch job called myjob.sh to the system

qsub myjob.sh



qtop

qtop is a scheduler command that provides a summary of all processes running on the cluster for a given user.

Examples

qtop is only available on the worker nodes. As such, you need to start an interactive session on a worker node usingqrsh or qrshx in order to use it.

To give a summary of all of your currently running jobs

qtop

Summary for job 256127

HOST VIRTUAL-MEM RSS-MEM %CPU %MEM CPUTIME+ COMMANDtestnode03 106.22 MB 1.79 MB 0.0 0.0 00:00:00 bashtestnode03 105.62 MB 1.27 MB 0.0 0.0 00:00:00 qtoptestnode03 57.86 MB 3.30 MB 0.0 0.0 00:00:00 ssh

--------- --------TOTAL: 0.26 GB 0.01 GB

Starting interactive jobs and submitting batch jobs

All job (both interactive sessions and batch jobs) on the University’s clusters are managed using the Son of Grid Enginejob scheduling software. You will typically see this referred to as SGE, as it is one of several derivatives of Sun GridEngine, or sometimes as just the scheduler.

SGE works ass follows: a user requests that a job (task), either a script or an interactive session, be run on the clusterand then SGE will take jobs from the queue based on a set of rules and priorities.

Interactive sessions

If you wish to use a cluster for interactive work, such as running applications like MATLAB or Ansys, or compilingsoftware, you will need to request interactive session from SGE. The pre-requisites for this (including connecting tothe clusters) are described in Using the HPC Systems.

There are three commands for requesting an interactive shell on a cluster worker node:

• qrsh - No support for graphical applications. Standard SGE command.

• qsh - Supports graphical applications. Standard SGE command.

• qrshx - Supports graphical applications. Superior to qsh. Unique to Sheffield’s clusters.

You can configure the resources available to the interactive session by specifying them as command line options to theqrshx, qsh or qrsh commands. For example to run a qrshx session with access to 16 GB of RAM:

[te1st@sharc-login2 ~]$ qrshx -l rmem=16G

or a session with access to 8 cores:

[te1st@sharc-login2 ~]$ qrshx -pe smp 8


https://arc.liv.ac.uk/trac/SGE

https://en.wikipedia.org/wiki/Oracle_Grid_Engine

https://en.wikipedia.org/wiki/Oracle_Grid_Engine


A table of Common Interactive Job Options is given below; any of these can be combined together to request moreresources.

Note: Long running jobs should use the batch submission system rather than requesting an interactive session for avery long time. Doing this will lead to better cluster performance for all users.

Common Interactive Job Options

Command Description-lh_rt=hh:mm:ss

Specify the total maximum execution time for the job. The upper limit is 08:00:00. NB theselimits may differ for specific SGE Projects/Queues.

-lrmem=xxG

Specify the maximum amount (xx) of (real) memory to be used (per process or core) inGigabytes.

-pe <env><nn>

Specify a parallel environment and a number of processor cores.

-pe smp<nn>

The smp parallel environment provides multiple threads on one node. <nn> specifies the maxnumber of threads.

Running batch jobs

The power of the clusters really comes from the batch job queue submission process. Using this system, you write ascript which requests various resources, initializes the computational environment and then executes your program(s).The scheduler will run your job when resources are available. As the task is running, the terminal output and anyerrors are captured and saved to disk, so that you can see the output and verify the execution of the task.

Any task that can be executed without any user intervention while it is running can be submitted as a batch job. Thisexcludes jobs that require a Graphical User Interface (GUI), however, many common GUI applications such as Ansysor MATLAB can also be used without their GUIs.

When you submit a batch job, you provide an executable file that will be run by the scheduler. This is normally a bashscript file which provides commands and options to the program you are using. Once you have a script file, or otherexecutable file, you can submit it to the queue by running:

qsub myscript.sh

Here is an example batch submission script that runs a fictitious program called foo:

#!/bin/bash# Request 5 gigabytes of real memory (mem)#$ -l rmem=5G

# load the module for the program we want to runmodule load apps/gcc/foo

# Run the program foo with input foo.dat# and output foo.resfoo < foo.dat > foo.res

Some things to note:

• The first line always needs to be #!/bin/bash (to tell the scheduler that this is a bash batch script).

• Comments start with a #



• Scheduler options, such as the amount of memory requested, start with #$

• You will often require one or more module commands in your submission file. These make programs andlibraries available to your scripts. Many applications and libraries are available as modules on ShARC andiceberg.

Here is a more complex example that requests more resources:

#!/bin/bash# Request 16 gigabytes of real memory (RAM)#$ -l rmem=16G# Request 4 cores in an OpenMP environment#$ -pe openmp 4# Email notifications to [email protected]#$ -M [email protected]# Email notifications if the job aborts#$ -m a

# Load the modules required by our programmodule load compilers/gcc/5.2module load apps/gcc/foo

# Set the OPENMP_NUM_THREADS environment variable to 4export OMP_NUM_THREADS=4

# Run the program foo with input foo.dat# and output foo.resfoo < foo.dat > foo.res



Scheduler Options

Command Description-lh_rt=hh:mm:ss

Specify the total maximum execution time for the job. The upper limit is typically 96:00:00 (4days) on ShARC and 168:00:00 (7 days) on Iceberg. Note that these limits may differ for specificSGE Projects/Queues. Also note that requesting less execution time may result in your jobspending less time queuing.

-lrmem=xxG

Specify the maximum amount (xx) of real memory to be used.

-l arch= Target a processor architecture. This is irrelevant on ShARC as all processors are the samemodel. Options on Iceberg include intel-e5-2650v2 and intel-x5650.

-N Job name, used to name output files and in the queue list.-jy[es]|n[o]

Join the error and normal output into one file rather than two.

-M Email address to send notifications to.-m bea Type of notifications to send. Can be any combination of begin (b) end (e) or abort (a) i.e. -m

ea for end and abortion messages.-a Specify the earliest time for a job to start, in the format MMDDhhmm. e.g. -a 01011130 will

schedule the job to begin no sooner than 11:30 on 1st January.-wdworking_dir

Execute the job from the directory specified (i.e. working_dir).

-lexcl=true

Request exclusive access to all nodes used by the job so no other jobs can run on them. This canbe useful for benchmarking purposes where you want to ensure that you have exclusive use ofe.g. memory/IO buses. Note that you still need to request CPU cores and memory to avoid beinglimited to just the default per job (one core and a set amount of RAM). Also note that the use ofthis option will likely result in longer queuing times.

-lhostname=

Target a node by name. Not recommended for normal use.

Frequently Asked SGE Questions

How many jobs can I submit at any one time

You can submit up to 2000 jobs to the cluster, and the scheduler will allow up to 200 of your jobs to run simultaneously(we occasionally alter this value depending on the load on the cluster).

How do I specify the processor type on Iceberg?

Add the following line to your submission script

#$ -l arch=intel-e5-2650v2

This specifies nodes that have the Ivybridge E5-2650 CPU. All such nodes on Iceberg have 16 cores.

To only target the older, 12 core nodes that contain X5650 CPUs add the following line to your submission script

#$ -l arch=intel-x5650

How do I specify multiple email addresses for job notifications?

Specify each additional email with its own -M option

#$ -M [email protected]#$ -M [email protected]


http://ark.intel.com/products/75269/Intel-Xeon-Processor-E5-2650-v2-20M-Cache-2_60-GHz

http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI


I want email notifications but don’t want to have to include my email address in every job submission script

Create a file called .sge_request in the directory you submit your jobs from containing:

-M [email protected] -M [email protected]

The -M parameter will be automatically supplied for all future job submissions. Note that you still need to requestemail notifications using -m (see above).

How do you ensure that a job starts after a specified time?

Add the following line to your submission script

#$ -a time

but replace time with a time in the format MMDDhhmm.

For example, for 22nd July at 14:10, you’d do

#$ -a 07221410

This won’t guarantee that it will run precisely at this time since that depends on available resources. It will, however,ensure that the job runs after this time. If your resource requirements aren’t too heavy, it will be pretty soon after.When I tried it, it started about 10 seconds afterwards but this will vary.

Getting started

See our guide to Starting interactive jobs and submitting batch jobs.

Submitting and controlling jobs programmatically

See our guide to the DRMAA API.

Reference information

Commands that allow you to interact with the scheduler:

• qhost - Show’s the status of Sun Grid Engine hosts.

• qrsh - Requests an interactive session on a worker node. No support for graphical applications.

• qrshx - Requests an interactive session on a worker node. Supports graphical applications. Superior to qsh inmost cases.

• qsh - Requests an interactive session on a worker node. Supports graphical applications.

• qstat - Displays the status of jobs and queues.

• qsub - Submits a batch job to the system.

• qtop - Provides a summary of all processes running on the cluster for a given user

Activating software using Environment Modules



Overview and rationale

‘Environment Modules’ are the mechanism by which much of the software is made available to the users of the ShARCand Iceberg clusters.

To make a particular piece of software available a user will load a module e.g. on ShARC, you can load a partic-ular version of the ‘scotch‘ library (version 6.0.4, built using the GCC 6.2 compiler and with support for parallelexecution using OpenMPI 2.0.1) with:

module load libs/scotch/6.0.4/gcc-6.2-openmpi-2.0.1

This command manipulates environment variables to make this piece of software available. If you then want to switchto using a different version of scotch (should another be installed on the cluster you are using) then you can run:

module unload libs/scotch/6.0.4/gcc-6.2-openmpi-2.0.1

then load the other.

You may wonder why modules are necessary: why not just install packages provided by the vender of the operatingsystem installed on the cluster? In shared high-performance computing environments such as ShARC and Iceberg:

• users typically want control over the version of applications that is used (e.g. to give greater confidence thatresults of numerical simulations can be reproduced);

• users may want to use applications built using compiler X rather than compiler Y as compiler X might generatefaster code and/or more accurate numerical results in certain situations;

• users may want a version of an application built with support for particular parallelisation mechanisms such asMPI for distributing work between machines, OpenMP for distributing work between CPU cores or CUDA forparallelisation on GPUs);

• users may want an application built with support for a particular library.

There is therefore a need to maintain multiple versions of the same applications on ShARC and Iceberg. Module filesallow users to select and use the versions they need for their research.

If you switch to using a cluster other than Iceberg or ShARC then you will likely find that environment modules areused there too. Modules are not the only way of managing software on clusters: increasingly common approachesinclude:

• the Conda package manager (Python-centric but can manage software written in any language; can be used onShARC and Iceberg);

• Singularity, a means for deploying software in containers (similar to Docker; can be used on ShARC).

Basic guide

You can list all (loaded and unloaded) modules on Iceberg or ShARC using:

module avail

You can then load a module using e.g.:

module load libs/geos/3.6.1/gcc-4.9.4

Note: Modules are not available on login nodes. You must start an interactive job on a worker node using qrshx,qsh or qrsh (see Using the HPC Systems) before any of the following commands will work.


https://en.wikipedia.org/wiki/Environment_variable

https://en.wikipedia.org/wiki/Operating-system-level_virtualization

https://www.docker.com/


You can then load further modules e.g.:

module load libs/gdal/2.2.0/gcc/gcc-4.9.4

Confirm which modules you have loaded using:

module list

If you want to stop using a module (by undoing the changes that loading that module made to your environment):

module unload libs/gdal/2.2.0/gcc/gcc-4.9.4

or to unload all loaded modules:

module purge

To learn more about what software is available on the system and discover the names of module files, you can viewthe online documentation for

• software on ShARC

• software on Iceberg

You can search for a module using:

module avail |& grep -i somename

The name of a Module should tell you:

• the type of software (application, library, development tool (e.g. compiler), parallel computing software);

• the name and version of the software;

• the name and version of compiler that the software was built using (if applicable; not all installed software wasinstalled from source);

• the name and version of used libraries that distinguish the different installs of a given piece of software (e.g. theversion of OpenMPI an application was built with).

Note that the module naming convention differs between ShARC and Iceberg.

Some other things to be aware of:

• You can load and unload modules in both interactive and batch jobs;

• Modules may themselves load other modules. If this is the case for a given module then it is typically noted inour documentation for the corresponding software;

• Available applications and application versions may differ between ShARC and Iceberg;

• The order in which you load modules may be significant (e.g. if module A sets SOME_ENV_VAR=apple andmodule B sets SOME_ENV_VAR=pear);

• Some related module files have been set up so that they are mutually exclusive e.g. on ShARC the modulesdev/NAG/6.0 and dev/NAG/6.1 cannot be loaded simultaneously (as users should never want to have bothloaded).

Behind the scenes

Let’s look at what happens when you load an enviroment. You can run the following example on ShARC (regardlessof whether the dev/NAG/6.1 module file loaded):



$ module show dev/NAG/6.1-------------------------------------------------------------------/usr/local/modulefiles/dev/NAG/6.1:

module-whatis Makes the NAG Fortran Compiler v6.1 availableconflict dev/NAGprepend-path PATH /usr/local/packages/dev/NAG/6.1/binprepend-path MANPATH /usr/local/packages/dev/NAG/6.1/mansetenv NAG_KUSARI_FILE /usr/local/packages/dev/NAG/license.lic

Here we see:

• The full path to the file that contains the definition of this module;

• A line briefly describing the purpose of the module (which could have been viewed separately using modulewhatis dev/NAG/6.1);

• An instruction not to load any other module files that start with dev/NAG as they will cause a conflict;

• A directory is prepended to the standard PATH variable: this ensures that executables relating to dev/NAG/6.1 are preferentially used unrelated executables in PATH directories that share the same filenames. Note thatthis directory is specific to this version (6.1) of the application we want to use;

• A directory is prepended to the standard MANPATH variable to ensure that the documentation (man pages) thatthe vendor bundled with the application can be found;

• An application-specific environment variable, NAG_KUSARI_FILE, is set (here to ensure that the applicationcan find a license file).

If you run the ‘env‘ command before and after loading a module you can see the effect of these changes.

Convenient ways to set up your environment for different projects

If you regularly need to activate multiple modules whilst working on a given project it may be tempting to add thenecessary module load commands to a shell startup script (e.g. the .bashrc script in your home directory).However, this is a Bad Idea for several reasons:

• Over time you will forget what is in your .bashrc and may forget that your workflow is dependent on modulesloaded by the script;

• Your .bashrc script may not be managed using version control (e.g. Git) or, if it is, it is unlikely to be in thesame repository as your project scripts/code;

• If someone asks you in three months’ time what version of an application you used to run a simulation will yoube able to tell them?

A better approach is to create a module-loading script inside the directory containing your project’s other scripts thensource (run) this script.

For example, you could have project scripts stored in a directory called /home/te1st/proj1.

You could create a script in that directory called setup_env.sh containing:

module load compilers/pgi/13.1module load mpi/pgi/openmpi/1.6.4

then if you want to load these modules in an interactive session or in a batch job you could run:

source /home/te1st/proj1/setup_env.sh


https://en.wikipedia.org/wiki/Man_page

https://git-scm.com/


If you want to run the job on both Iceberg and ShARC (which provide different software / module files) you couldadapt your script to load different modules depending on the cluster name e.g.

case $SGE_CLUSTER_NAME iniceberg)

module load compilers/pgi/13.1module load mpi/pgi/openmpi/1.6.4;;

sharc)module load mpi/openmpi/2.0.1/pgi-17.5;;

esac

Managing your environment this way is more likely to result in reproducible research, particularly if changes to thecontent of /home/te1st/proj1 are tracked using Git or another version control tool

Managing your own module files

Modules are a great way of loading/unloading software installed in non-standard places. You may therefore want touse them to manage software installed in

• your home directory

• a directory shared by your research group

If you want your own Modules, you typically need to create a hierarchy of directories and files. Within a basedirectory the relative path to a given module file determines the name you need to use to load it. See the /usr/local/modulefiles directories on ShARC and Iceberg to:

• see the files that provide all cluster-wide modules and

• get an understanding of the (Tcl) syntax and structure of module files.

A tutorial on how to write module files is not provided here (but may be in future).

Once you’ve created a set of module files within a directory you can make the module system aware of them byrunning:

module use /the/path/to/my/modules

The next time you run module avail you will see that your modules are listed alongside the cluster-wide modules.

If you no longer want to to have access to your own module files then you can run:

module unuse /the/path/to/my/modules

Module Command Reference

Here is a list of the most useful module commands. For full details, type man module at the command prompt onone of the clusters.

• module list – lists currently loaded modules

• module avail – lists all available modules

• module load modulename – loads module modulename

• module unload modulename – unloads module modulename

• module switch oldmodulename newmodulename – switches between two modules


https://www.tcl.tk/


• module show modulename - Shows how loading modulename will affect your environment

• module purge – unload all modules

• module help modulename – may show longer description of the module if present in the modulefile

• man module – detailed explanation of the above commands and others

More information on the Environment Modules software can be found on the project’s site.

Jupyter and JupyterHub

Introduction

Jupyter Notebooks are executable documents containing: formatted text, formatted maths, chunks of code plusfigures, tables and textual output generated by that code.

Notebooks can be used:

• to develop and execute linear data analysis workflows;

• to present linear workflows to others;

• as runnable documentation for software packages (e.g. NetworkX examples);

• as executable teaching materials (e.g. BAD days tutorials);

• as runnable journal papers?

Jupyter itself is web application that interprets, runs and renders Notebooks. You interact with it by As you interactwith it by just connecting from your web browser the Jupyter server software can be running on your local machine ora remote server (which may have more memory, CPU cores and/or GPUs than your local machine).

On the university’s ShARC cluster a (beta) JupyterHub service allows a user to:

1. Log in to the JupyterHub web inteface,

2. Specify what resources (memory, CPU cores, GPUs) they want for a Jupyter session,

3. Start and run a Jupyter Notebook server on a worker node in the cluster using these resources.

Using Jupyter on ShARC

JupyterHub on ShARC: connecting, requesting resources and starting a session

Connecting

1. From a web browser navigate to the following URL from a machine that is on-campus or has a VPN connectionto the University:

https://jupyter-sharc.shef.ac.uk

2. When prompted, log in using your ShARC/Iceberg username (e.g. te1st, not the first part of your emailaddress) and password.


http://modules.sourceforge.net/

https://jupyter.org/

http://nbviewer.jupyter.org/github/machinalis/satimg/blob/master/Searching%20for%20aliens.ipynb

https://networkx.github.io/documentation/stable/auto_examples/index.html

http://bitsandchips.me/BAD_days/

http://nbviewer.jupyter.org/github/cossatot/lanf_earthquake_likelihood/blob/master/notebooks/lanf_manuscript_notebook.ipynb

https://jupyterhub.readthedocs.io/




3. Click Start server when prompted:

You can only run one Jupyer Notebook Server on the cluster at a time but you can use your Notebook Server torun multiple Notebooks.

Requesting resources (Spawner options)

Before the Notebook server starts you may wish to request resources such as memory, multiple CPUs and GPUs usingthe Spawner options form. It presents you with a number of options:

Project

As a user on the cluster you are a member of one or more User Groups, which in turn give access to Projects. Bysubmitting a job using a Project you can:

• Use restricted-access worker nodes (possibly bought by your research group);

• Have your job’s CPU and memory usage logged in a way that ensures all users of the cluster get fair usage.

Most users do not need to select a specific Project and should leave this setting as its default.



Currently supported projects:

• gpu: GPU-equipped nodes (public)

• rse: The Research Software Engineering group’s nodes (restricted access)

• cstest: CiCS testing only

Support for other Projects can be added on request.

Job Queue

Selecting any lets the scheduler choose an appropriate Job Queue, which is typically what you want.

Support for additional Job Queues can be added on request.

Email address

The resource (CPU, memory and runtime) usage of your Jupyter session will be emailed to this address when it isstopped or aborts.

This information can by useful for diagnosing why the job scheduler has killed your Jupyter session should yoursession use more computation resources than you requested at the outset.

CPU cores

If you select >1 core then you must also select a Parallel Environment otherwise only 1 core will be granted.

Parallel Environment

• smp: The number of CPU cores selected above are all allocated on one node.

• mpi: The number of CPU cores selected above are allocated on one or more nodes.

RAM per CPU core

A value in gigabytes.

GPUS per CPU core

Requires that Project is gpu (public GPUs) or rse (private GPUs).

Notebook session runtime

This is currently fixed at 4 hours.


http://rse.shef.ac.uk


Starting your Notebook session using the requested resources

After you’ve specified the resources you want for your job, click Spawn to try starting a Jupyter session on one (ormore) worker nodes.

This may take a minute.

Warning: If the cluster is busy or you have requested an incompatible or otherwise unsatisfiable set of resourcesfrom the job scheduler then this attempt to start a session will time out and you will return to the Spawner optionsform.

Once your session has started you should see the Jupyter file browser interface:

Jupyter’s file browser

After starting a Jupyter session you are presented with Jupyter’s file browser view, which is the default tab in Jupyter’suser interface. This view shows you files on the machine running your Jupyter session (here, the cluster), not yourlocal machine. This behaves much like a desktop file browser application:



• Click on a directory to browse into it.

• Click on a text file (e.g. a R or Python script) to view/edit it in your browser. You’ll notice that Jupyter’s texteditor understands different programming languages and will use colour to make key syntactic elements easierto identify.

• Click on a Notebook (.ipynb filename suffix) to open it in a new browser tab.

• Rename, delete or move files/folders by checking the relevant tick-boxes to the left of each item of interest thenclick the appropriate option above.

• The drop-down menu at the top of the column of check-boxes allows you to filter to view just Notebooks, allNotebooks that are currently running, all files or all folders.

Warning: Certain directories are may not be accessible via this interface:

• /home/username

• /data/username

• /shared/volname

This set of directories are automounted i.e. made available to the user on demand but you cannot express thatdemand via this interface. If you browse into /data and it is empty or does not contain your personal subdirectorythen you need to briefly open a Jupyter terminal and run:

ls /data/username

then that directory should subsequently be visible/accessible in this file browser.

Jupyter’s web terminal

You can start a terminal from the New menu within Jupyter’s Files tab. This is a fully-featured web terminal, runningon a worker node. It opens a new browser tab.



You can use this terminal to perform any command-line-only operation on the cluster, including:

• Managing versions of Notebooks / other code using version control software such as git (as illustrated above);

• Searching for files/directories;

• Configuring Python or R environments (although there is an (enabled) plug-in for creating/selecting environ-ments and installing/upgrading/removing packages using Jupyter’s graphical interface);

• Triggering the automounting of directories not visible in JupyterHub’s file browser.

Having a terminal interface available within the browser negates the need to separately log into the cluster via SSH toaccess the command-line.

Limitations:

• Not all keyboard shortcuts that typically work in terminal programs work within this web console:

– tab can be used for tab completion

– but Ctrl-C kills the program running in the terminal

– To copy text, select it, right-click and click Copy.

– To paste text, right-click and click Paste.

• You cannot start graphical programs (e.g. xeyes) from this terminal.

Jupyter on SHARC: preparing your environment

Background

Once you have a Jupyter Notebook server running (e.g. on a cluster worker node) you typically want to consider yourexecution environment, which is your choice of

• kernel (code cell language)

• software packages/libraries

Kernels

Jupyter Notebooks were originally known as IPython Notebooks but now Jupyter supports Notebook code cells writtenin many different languages. This is achieved by the (per-user) Jupyter Notebook server sending the contents of codecells to a kernel for evaluation. The most widely used kernels are:

• the IPython kernel, the default kernel, which can run code cells containing Python >= 3.3 and Python 2.7;


https://swcarpentry.github.io/git-novice/

https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

https://github.com/ipython/ipykernel


• IRKernel, a kernel for R >= 3.2.

Packages

Most notebooks make use of external software packages for e.g. fitting statistical models to data. There are severaldifferent ways you might enable/load packages on ShARC (including module files and Singularity containers) but ifusing Jupyter it is recommended that you install and activate software using the conda package manager if possible.This can be done using Jupyter’s graphical interface (in most cases) or from the command-line.

Environments

conda packages are installed into environments. An environment is an isolated collection of packages. You mightcreate:

• One environment for one project containing Python 3, the IPython kernel and the pandas and matplotlibPython packages (plus dependencies) for data analysis.

• Another environment for a second project containing R, the IRKernel and the dplyr and gplot2 R packages.

• A third environment containing Python 2.7 plus pandas but no kernel for work that doesn’t involve JupyterNotebooks.

conda allows users to

• Install and manage packages without a system adminstrator needing to be involved;

• Isolate and audit the set of packages used per project (good for reproducible research)

• Share environment definitions with others and with automated test/build systems (i.e. continuous integration)

Using conda on ShARC via Jupyter

From the browser tab containing the Jupyter’s file browser activate the Conda tab within Jupyter’s interface.

You should then see something like:


https://irkernel.github.io/

https://www.r-project.org/

https://conda.io/docs/using/envs.html



https://en.wikipedia.org/wiki/Continuous_integration


Here we have (in anticlockwise order) lists of:

1. All conda environments that have been found and can be activated from within Jupyter;

2. Packages (inc. versions) that can be installed into a selected environment; by default only the latest versionfrom the default conda channel (package repository) can be installed this way.

3. Packages that are installed in the environment selected in the above pane (plus the package version and thebuild version (e.g. version of Python it was built for)).

These three views may take a few seconds to populate after clicking Conda.

Warning: This interface currently does not reliably allow conda environments to be created or modified.Until this issue is resolved you can use this interface to inspect your environments but you should create andmodify conda environments from a terminal.

Creating a new conda environment

Before we run a Notebook we typically want to create a new conda environment containing the packages we areinterested in plus a Jupyter kernel.

See the general documentation for using conda on ShARC for generic instructions on how to create conda environmentsfrom the command-line. Note that if you are using a Jupyter Terminal then you do not need to load conda usingmodule load .... Make sure you install a package containing a Jupyter kernel (e.g. ipykernel for Pythonwork) into your environment.




When following that documentation you might want to use the following as starting points for creating Jupyter(Hub)-compatible environments:

Python 3:

conda create -n example-python-env python=3.6 ipykernel jupyter_client

R:

conda create -n example-r-env python=3.6 r-irkernel jupyter_client libiconf

Python from the Intel Python Distribution <https://software.intel.com/en-us/distribution-for-python>:

conda create -n example-intel-python-env -c intel intelpython3_core ipykernel jupyter_→˓client

Capturing the state of an environment

It is important to track the versions of packages you used to generate your research outputs, primarily to allow you andothers to easily repeat your workflows. Ideally you should manage a file detailing the packages in your environment,plus your Notebook and other project files, in a version control system such as git.

After creating/modifying an environment:

• Click on the export icon (left-most icon beneath Action) for a given environment to download a conda environ-ment definition file.

• Alternatively you can generate a definition file from a Jupyter Terminal:

source activate my-env-namecd /data/$USER/research-project-7conda env export > environment.yml

Next

After you have assessed what environments you have available, you can start creating, editing and running JupyterNotebooks.

Creating, editing and running Jupyter Notebooks

Creating Notebooks

After creating or deciding on a conda environment that you want to run a Notebook in you can now create a Notebookor open an existing one.

To create a Notebook:

1. Return to the Jupyter Home browser tab;

2. Click the Files Jupyter tab;

3. Browse to the directory where you want to create your new Notebook;


https://en.wikipedia.org/wiki/Git


4. Click New then (beneath Notebooks) the name of the conda environment you wish to use (e.g.rdkit-sharc). Do not use the jupyterhub or jupyterhub-dev environments. You are advised notto use the anaconda environments either as these are read-only to most users and users have little controlover if/when they are updated and what packages they contain. It is recommended that you create your ownenvironments.

5. A blank Notebook should appear in a new browser tab.

Your Notebook will have access to the packages installed in the selected environment.

Opening existing Notebooks

Alternatively you can click on an existing Notebook (.ipynb) file in Jupyter’s file browser to open it.



Using Jupyter Notebooks

The basics of using Jupyter Notebooks to create self-describing, runable workflow documents are explained in theJupyter Notebook official documentation.

Monitoring and controlling your Jupyter session

Monitoring and control

If you want to see what Jupyter Notebooks and Terminals your Jupyter server is running then

1. Click the Home browser tab;

2. Click the Running browser tab.

Here you can shut down (close) individual Notebooks, which may be useful to free up the memory and CPU coresthat the cluster’s job scheduler has allocated for your JupyterHub session.


http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Notebook%20Basics.html


If you want to stop your entire Jupyter server (i.e. end your JupyterHub session) you can

1. Click Control Panel on the Home browser tab;

2. Click Stop My Server

Your JupyterHub session may terminate for other reasons:

• The cluster’s job scheduler may stop your JupyterHub job if your Jupyter server exceeds the amount of RAMyou requested via the Spawner Options page.

• Your JupyterHub job has been running for longer than the (fixed) duration specified on the Spawner Optionspage.

—

Session persistence

You can close your Jupyter-related browser tabs and your Jupyter session will keep running until it terminates forone of the reasons listed above.

You can then revisit the JupyterHub site to reconnect to your Jupyter session and carry on from where you left off.

Warning: If you leave your Jupyter session running and are not using it then you are using CPU cores, memoryand possibly GPUs that cannot be used by others.

Please stop your Jupyter session if you no longer need it.

JupyterHub: errors and troubleshooting

Common issues

After submitting Spawner options I get returned to the Spawner options page after a minute or two

The cluster is not able to start your JupyterHub session at this time because:



• the cluster is currently doesn’t have the capacity to start your JupyterHub job on demand at this time or

• you have requested an incompatible set of resources from the Spawner options form or

• the job scheduler allocates you resources on cluster nodes but the Jupyter server software cannot start for somereason.

To determine if the third of these possibilities is the case check the most recent jupyterhub.oXXXXXX log file inyour home directory for error messages.

I encounter errors when trying to create/modify environments via the Conda tab in Jupyter

This is a known issue that will hopefully be resolved in future. For now create and modify conda environments fromthe command-line.

I cannot browse to subdirectories below /data or /shared from the Jupyter file browser

This is a known issue; a workaround is described here.

Less common issues

I get a ‘503’ error after logging in to JupyterHub

If you modify the PYTHON_PATH variable in your .bashrc file your Jupyter server may not start correctly. Thesolution to this is to remove these lines from your .bashrc file.

Also,f you have previously tried installing and running Jupyter yourself (i.e. not using this JupyterHub interface) thenyou may get 503 errors when connecting to JupyterHub due to the old .jupyter profile in your home directory; if youthen find a Jupyter log file in your home directory containing SSL WRONG_VERSION_NUMBER then try deleting (orrenaming) the .jupyter directory in your home directory.

Status of and maintenance of ShARC’s JupyterHub service

This service is currenty experimental. If you use this service and encounter a problem, please provide feedback [email protected].

The server that provider the JupyterHub service is typically rebooted at 03:26 on the 2nd Tuesday of the month toinstall security updates.

JupyterHub on a Grid Engine cluster: internal workings

The hub of JupyterHub has several components:

• an authenticator that allows users to log in, possibly using externally-stored identity information;

• a database of user and state information;

• a spawner that can start single-user Jupyter Notebook servers on demand.


https://github.com/jupyter/jupyterhub/issues/294



There is also a web proxy that first routes web connections from a given user to the hub for authentication andpossibly choosing spawner options then, after a single-user Jupyter server has been spawned, certain web connectionsare forwarded to the Jupyter Notebook server. From the user’s perspective it appears that they are interacting with asingle web application, even though at times they might be talking to a single-user Jupyter server that running on adifferent machine to the Hub.

ShARC uses a custom spawner, sgespawner, that spawns single-user Jupyter servers on one or more worker nodes onShARC by submitting batch jobs to the Grid Engine job scheduler.

The JupyterHub and sgespawner configuration allows the user to specify the Grid Engine resources required forthe Jupyter session in advance via a web form then these resources are requested as part of the batch job submission.

Further details of how JupyterHub and sgespawner are configured on ShARC can be found in this repository.

Credits

The JupyterHub service on ShARC is currently developed and maintained by the University’s Research SoftwareEngineering team. This work has been funded by OpenDreamKit, a Horizon2020 European Research Infrastructure

project (676541) that aims to advance the open source computational mathematics ecosystem.

Using Jupyter on Iceberg

From a web browser navigate to:

https://jupyter.shef.ac.uk

The JupyterHub service for Iceberg predates that of ShARC and is much more basic. There is currently no way youcan request specific resources (multiple CPU cores, GPU(s), more RAM etc) from the cluster’s job scheduler whenusing JupyterHub. Your Notebook session will therefore get one CPU core and the default amount of RAM per jobfor the cluster you’ve connected to.


https://github.com/willfurnass/sgespawner

https://github.com/RSE-Sheffield/jupyterhub-gridengine-sharc/



http://opendreamkit.org/

https://ec.europa.eu/programmes/horizon2020/

https://ec.europa.eu/programmes/horizon2020/en/h2020-section/european-research-infrastructures-including-e-infrastructures

http://cordis.europa.eu/project/rcn/198334_en.html

http://opendreamkit.org/


Iceberg

Software on iceberg

These pages list the software available on iceberg. If you notice an error or an omission, or wish to request newsoftware please email the research computing team at [email protected].

There are many versions of applications, libraries and compilers installed on the iceberg cluster. In order to avoidconflict between these software items we employ a system called modules. Having logged onto a worker nodes, usersare advised to select the software (and the version of the software) they intend to use by using the module command.

Applications on Iceberg

CASTEP

CASTEP

Latest Version 16.11

URL http://www.castep.org/

Licensing

Only licensed users of CASTEP are entitled to use it and license details are available on CASTEP’s website. Accessto CASTEP on the system is controlled using a Unix group. That is, only members of the castep group can accessand run the program. To be added to this group, you will need to contact [email protected] andprovide evidence of your eligibility to use CASTEP.

Interactive Usage

The serial version of CASTEP should be used for interactive usage. After connecting to iceberg (see Establishing aSSH connection), start an interactive session with the qrsh or qsh command. Make the serial version of CASTEPavailable using one of the following commands:

module load apps/intel/15/castep/16.11-serialmodule load apps/intel/15/castep/16.1-serialmodule load apps/intel/15/castep/8.0-serial

The CASTEP executable is called castep.serial so if you execute:

castep.serial

You should get the following:

Usage:castep <seedname> : Run files <seedname>.cell [and <seedname>.param]

" [-d|--dryrun] <seedanme> : Perform a dryrun calculation on files <seedname>.→˓cell" [-s|--search] <text> : print list of keywords with <text> match in

→˓description

2.2. Iceberg 41


http://www.castep.org/

http://www.castep.org/CASTEP/GettingCASTEP


" [-v|--version] : print version information" [-h|--help] <keyword> : describe specific keyword in <>.cell or <>.param" " all : print list of all keywords" " basic : print list of basic-level keywords" " inter : print list of intermediate-level keywords" " expert : print list of expert-level keywords" " dummy : print list of dummy keywords

If, instead, you get:

-bash: castep.serial: command not found

It is probably because you are not a member of the castep group. See Licensing for details on how to be added tothis group.

Interactive usage is fine for small CASTEP jobs such as the Silicon example given at http://www.castep.org/Tutorials/BasicsAndBonding

To run this example, you can do:

# Get the files, decompress them and enter the directory containing themwget http://www.castep.org/files/Si2.tgztar -xvzf ./Si2.tgzcd Si2

# Run the CASTEP job in serialcastep.serial Si2

# Read the output using the more commandless Si2.castep

CASTEP has a built in help system. To get more information on using castep use:

castep.serial -help

Alternatively you can search for help on a particular topic:

castep.serial -help search keyword

or list all of the input parameters:

castep.serial -help search all

Batch Submission - Parallel

The parallel version of CASTEP is called castep.mpi. To make the parallel environment available, use one of thefollowing module commands:

module load apps/intel/15/castep/16.11-parallelmodule load apps/intel/15/castep/16.1-parallelmodule load apps/intel/15/castep/8.0-parallel

As an example of a parallel submission, we will calculate the bandstructure of graphite following the tutorial athttp://www.castep.org/Tutorials/BandStructureAndDOS

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qrsh or qshcommand. Download and decompress the example input files with the commands


http://www.castep.org/Tutorials/BasicsAndBonding


http://www.castep.org/Tutorials/BandStructureAndDOS


wget http://www.castep.org/files/bandstructure.tgztar -xvzf ./bandstructure.tgz

Enter the directory containing the input files for graphite:

cd bandstructure/graphite/

Create a file called submit.sge that contains the following:

#!/bin/bash#$ -pe openmpi-ib 4 # Run the calculation on 4 CPU cores#$ -l rmem=4G # Request 4 GB of real memory per coremodule load apps/intel/15/castep/16.11-parallel

mpirun castep.mpi graphite

Submit it to the system with the command:

qsub submit.sge

After the calculation has completed, get an overview of the calculation by looking at the file graphite.castep:

more graphite.castep

Installation Notes

These are primarily for system administrators.

Version 16.11

Serial (no MPI) and parallel (MPI) builds were compiled. Both builds were compiled with Intel compiler 15.0.3(including the Intel MKL 2015.3 for BLAS and FFT routines). The parallel build was compiled using OpenMPI1.10.0.

Both builds were installed using this script Note that this compiles both builds in /data/$USER as the builddirectory must be availble to all cluster nodes to allow for subsequent Testing of the parallel build. ~2.2 GB of freespace is required.

• The non-MPI build modulefile was installed as /usr/local/modulefiles/apps/intel/15/castep/16.11-serial

• The MPI build modulefile was installed as /usr/local/modulefiles/apps/intel/15/castep/16.11-parallel

Version 16.1

The jump in version numbers from 8 to 16.1 is a result of CASTEP’s change of version numbering. There are noversions 9-15.

Serial (1 CPU core) and Parallel versions of CASTEP were compiled. Both versions were compiled with version15.0.3 of the Intel Compiler Suite and the Intel MKL versions of BLAS and FFT were used. The parallel version madeuse of OpenMPI 1.8.8

The Serial version was compiled and installed with

2.2. Iceberg 43


module load compilers/intel/15.0.3install_dir=/usr/local/packages6/apps/intel/15/castep/16.1mkdir -p $install_dir

tar -xzf ./CASTEP-16.1.tar.gzcd CASTEP-16.1

#Compile Serial versionmake INSTALL_DIR=$install_dir FFT=mkl MATHLIBS=mkl10make INSTALL_DIR=$install_dir FFT=mkl MATHLIBS=mkl10 install install-tools

The directory CASTEP-16.1 was then deleted and the parallel version was installed with

#!/bin/bashmodule load libs/intel/15/openmpi/1.8.8#The above command also loads Intel Compilers 15.0.3#It also places the MKL in LD_LIBRARY_PATH

install_dir=/usr/local/packages6/apps/intel/15/castep/16.1


#Workaround for bug described at http://www.cmth.ph.ic.ac.uk/computing/software/→˓castep.htmlsed 's/-static-intel/-shared-intel/' obj/platforms/linux_x86_64_ifort15.mk -i

#Compile parallel versionmake COMMS_ARCH=mpi FFT=mkl MATHLIBS=mkl10mv ./obj/linux_x86_64_ifort15/castep.mpi $install_dir

CASTEP Version 8

Serial (1 CPU core) and Parallel versions of CASTEP were compiled. Both versions were compiled with version15.0.3 of the Intel Compiler Suite and the Intel MKL versions of BLAS and FFT were used. The parallel version madeuse of OpenMPI 1.8.8

The Serial version was compiled and installed with

module load compilers/intel/15.0.3install_dir=/usr/local/packages6/apps/intel/15/castep/8.0


#Compile Serial versionmake INSTALL_DIR=$install_dir FFT=mkl MATHLIBS=mkl10make INSTALL_DIR=$install_dir FFT=mkl MATHLIBS=mkl10 install install-tools

The directory CASTEP-8.0 was then deleted and the parallel version was installed with

#!/bin/bashmodule load libs/intel/15/openmpi/1.8.8#The above command also loads Intel Compilers 15.0.3#It also places the MKL in LD_LIBRARY_PATH

install_dir=/usr/local/packages6/apps/intel/15/castep/8.0mkdir -p $install_dir




#Compile parallel versionmake COMMS_ARCH=mpi FFT=mkl MATHLIBS=mkl10mv ./obj/linux_x86_64_ifort15/castep.mpi $install_dir

Modulefiles

• CASTEP 16.11-serial

• CASTEP 16.11-parallel





Testing

Version 16.1 Serial

The following script was submitted via qsub from inside the build directory:

#!/bin/bash#$ -l rmem=10Gmodule load compilers/intel/15.0.3

cd CASTEP-16.1/Test../bin/testcode.py -q --total-processors=1 -e /home/te1st/CASTEP/CASTEP-16.1/obj/→˓linux_x86_64_ifort15/castep.serial -c simple -v -v -v

All but one of the tests passed. It seems that the failed test is one that fails for everyone for this version since there isa missing input file. The output from the test run is on the system at /usr/local/packages6/apps/intel/15/castep/16.1/CASTEP_SERIAL_tests_09022016.txt

Version 16.1 Parallel

The following script was submitted via qsub from inside the build directory

#!/bin/bash#$ -pe openmpi-ib 4#$ -l rmem=10Gmodule load libs/intel/15/openmpi/1.8.8

cd CASTEP-16.1/Test../bin/testcode.py -q --total-processors=4 --processors=4 -e /home/te1st/CASTEP/→˓CASTEP-16.1/obj/linux_x86_64_ifort15/castep.mpi -c simple -v -v -v

All but one of the tests passed. It seems that the failed test is one that fails for everyone for this version since there isa missing input file. The output from the test run is on the system at /usr/local/packages6/apps/intel/15/castep/16.1/CASTEP_Parallel_tests_09022016.txt

Version 8 Parallel The following script was submitted via qsub

2.2. Iceberg 45


#!/bin/bash#$ -pe openmpi-ib 4module load libs/intel/15/openmpi/1.8.8

cd CASTEP-8.0make check COMMS_ARCH=mpi MAX_PROCS=4 PARALLEL="--total-processors=4 --processors=4"

All tests passed.

GATK

GATK

Version 3.4-46

URL https://www.broadinstitute.org/gatk/

The Genome Analysis Toolkit or GATK is a software package for analysis of high-throughput sequencing data, devel-oped by the Data Science and Data Engineering group at the Broad Institute. The toolkit offers a wide variety of tools,with a primary focus on variant discovery and genotyping as well as strong emphasis on data quality assurance. Itsrobust architecture, powerful processing engine and high-performance computing features make it capable of takingon projects of any size.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive sesssion with the qsh or qrshcommand.

The latest version of GATK (currently 3.4-46) is made available with the command

module load apps/binapps/GATK

Alternatively, you can load a specific version with

module load apps/binapps/GATK/3.4-46module load apps/binapps/GATK/2.6-5

Version 3.4-46 of GATK also changes the environment to use Java 1.7 since this is required by GATK 3.4-46. Anenvironment variable called GATKHOME is created by the module command that contains the path to the requestedversion of GATK.

Thus, you can run the program with the command

java -jar $GATKHOME/GenomeAnalysisTK.jar -h

Which will give a large amount of help, beginning with the version information

---------------------------------------------------------------------------------The Genome Analysis Toolkit (GATK) v3.4-46-gbc02625, Compiled 2015/07/09 17:38:12Copyright (c) 2010 The Broad InstituteFor support and documentation go to http://www.broadinstitute.org/gatk---------------------------------------------------------------------------------


https://www.broadinstitute.org/gatk/


Documentation

The GATK manual is available online https://www.broadinstitute.org/gatk/guide/

Installation notes

The entire install is just a .jar file. Put it in the install directory and you’re done.

Modulefile

Version 3.4-46

• The module file is on the system at /usr/local/modulefiles/apps/binapps/GATK/3.4-46

Its contents are

#%Module1.0######################################################################### GATK 3.4-46 modulefile##

## Module file loggingsource /usr/local/etc/module_logging.tcl##

#This version of GATK needs Java 1.7module load apps/java/1.7

proc ModulesHelp { } {puts stderr "Makes GATK 3.4-46 available"

}

set version 3.4-46set GATK_DIR /usr/local/packages6/apps/binapps/GATK/$version

module-whatis "Makes GATK 3.4-46 available"

prepend-path GATKHOME $GATK_DIR

Version 2.6-5

• The module file is on the system at /usr/local/modulefiles/apps/binapps/GATK/2.6-5

Its contents are

#%Module1.0######################################################################### GATK 2.6-5 modulefile##


proc ModulesHelp { } {puts stderr "Makes GATK 3.4-46 available"

2.2. Iceberg 47

https://www.broadinstitute.org/gatk/guide/


}

set version 2.6.5set GATK_DIR /usr/local/packages6/apps/binapps/GATK/$version

module-whatis "Makes GATK 2.6-5 available"

prepend-path GATKHOME $GATK_DIR

MOMFBD

MOMFBD

Version 2016-04-14

URL http://dubshen.astro.su.se/wiki/index.php?title=MOMFBD

MOMFBD or Multi-Object Multi-Frame Blind Deconvolution is a image processing application for removing theeffects of atmospheric seeing from solar image data.

Usage

The MOMFBD binaries can be added to your path with:

module load apps/gcc/5.2/MOMFBD

Installation notes

MOMFBD was installed using gcc 5.2. Using this script and is loaded by this modulefle.

STAR

STAR

Latest version 2.5.0c

URL https://github.com/alexdobin/STAR

Spliced Transcripts Alignment to a Reference.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrsh command.

The latest version of STAR (currently 2.5.0c) is made available with the command


http://dubshen.astro.su.se/wiki/index.php?title=MOMFBD

https://github.com/alexdobin/STAR


module load apps/gcc/5.2/STAR


module load apps/gcc/5.2/STAR/2.5.0c

This command makes the STAR binary available to your session and also loads the gcc 5.2 compiler environmentwhich was used to build STAR. Check that STAR is working correctly by displaying the version

STAR --version

This should give the output

STAR_2.5.0c

Documentation

The STAR manual is available online: https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf

Installation notes

STAR was installed using gcc 5.2

module load compilers/gcc/5.2

mkdir STARmv ./STAR-2.5.0c.zip ./STARcd ./STAR/unzip STAR-2.5.0c.zipcd STAR-2.5.0cmakecd sourcemkdir -p /usr/local/packages6/apps/gcc/5.2/STAR/2.5.0cmv ./STAR /usr/local/packages6/apps/gcc/5.2/STAR/2.5.0c/

Testing

No test suite was found.

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/STAR/2.5.0c

It’s contents

#%Module1.0######################################################################### STAR 2.5.0c module file##

## Module file logging

2.2. Iceberg 49

https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf


source /usr/local/etc/module_logging.tcl##


module-whatis "Makes version 2.5.0c of STAR available"

set STAR_DIR /usr/local/packages6/apps/gcc/5.2/STAR/2.5.0c/

prepend-path PATH $STAR_DIR

Abaqus

Abaqus

Versions 2017, 6.14, 6.13, 6.12 and 6.11

Support Level FULL

Dependancies Intel Compiler (for user subroutines)

URL http://www.3ds.com/products-services/simulia/products/abaqus/

Local URL https://www.sheffield.ac.uk/cics/research/software/abaqus

Abaqus is a software suite for Finite Element Analysis (FEA) developed by Dassault Systèmes.

Interactive usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh command.Alternatively, if you require more memory, for example 16 gigabytes, use the command qsh -l rmem=16G

Make a specific version available with one of the following commands:

module load apps/abaqus/2017/binarymodule load apps/abaqus/614module load apps/abaqus/613module load apps/abaqus/612module load apps/abaqus/611

Command-line access

To access Abaqus’ command-line interface:

abaqus

Graphical interface

The recommended way to start Abaqus’ graphical interface depends on

• which version of Abaqus has been loaded


http://www.3ds.com/products-services/simulia/products/abaqus/

https://www.sheffield.ac.uk/cics/research/software/abaqus


• whether you are using a (hardware-accelerated) ‘qsh-vis’ session which can greatly improve graphical per-formance but requires a graphics card, which are typically in demand.

Without qsh-vis (> 6.13)

abaqus cae -mesa

With qsh-vis (<= 2017)

abaqus cae

With qsh-vis (> 6.13 and < 2017)

$ABAQUSCOMMAND cae

With qsh-vis (6.13)

abq6133 cae

Example problems

Abaqus contains a large number of example problems which can be used to become familiar with Abaqus on thesystem. These example problems are described in the Abaqus documentation, and can be obtained using the Abaqusfetch command. For example, after loading the Abaqus module enter the following at the command line to extractthe input file for test problem s4d:

abaqus fetch job=s4d

This will extract the input file s4d.inp. To run the computation defined by this input file replaceinput=myabaqusjob with input=s4d in the commands and scripts below.

Batch jobs

Single-core job

In this example, we will run the s4d.inp file on a single core using 8 Gigabytes of memory. After connecting toiceberg (see Establishing a SSH connection), start an interactive sesssion with the qrsh command.

Load version 2017 of Abaqus and fetch the s4d example by running the following commands:

module load apps/abaqus/2017/binaryabaqus fetch job=s4d

Now, you need to write a batch submission file. We assume you’ll call this my_job.sge:

2.2. Iceberg 51


#!/bin/bash#$ -cwd#$ -l rmem=8G

module load apps/abaqus/2017/binary

abaqus job=my_job input=s4d.inp scratch=$TMPDIR memory="8gb" interactive

Submit the job with:

qsub my_job.sge

• We have requested 8 gigabytes of memory in the above job. The memory="8gb" switch tells abaqus to use 8gigabytes. The #$ -l rmem=8G tells the system to reserve 8 gigabytes of real memory. Make sure that thememory= and rmem= values match.

• Note the word interactive at the end of the abaqus command. Your job will not run without it.

Single-core job with user subroutine

In this example, we will fetch a simulation from Abaqus’ built in set of problems that makes use of user subroutines(UMATs) and run it in batch on a single core. After connecting to iceberg (see Establishing a SSH connection), startan interactive session with the qrsh command.

Load version 2017 of Abaqus and fetch the umatmst3 example by running the following commands:

module load apps/abaqus/2017/binaryabaqus fetch job=umatmst3*

This will produce two files:

• The input file umatmst3.inp

• the Fortran user subroutine umatmst3.f

Now, you need to write a batch submission file. We assume you’ll call this my_user_job.sge:

#!/bin/bash#$ -cwd#$ -l rmem=8G

module load apps/abaqus/2017/binarymodule load $ABAQCOMPVER

abaqus job=my_user_job input=umatmst3.inp user=umatmst3.f scratch=$TMPDIR memory="8gb→˓" interactive

Submit the job with:

qsub my_user_job.sge

Important notes:

• In order to use user subroutines, it is necessary to load the module for a particular version of the Intel compiler.The name of the module file for the most appropriate Intel compiler is stored in the ABAQCOMPVER environmentvariable.

• The user subroutine itself is passed to Abaqus with the switch user=umatmst3.f.



• The notes for the previous single-core batch job example still apply.

Multi-core job (on a single node)

To distribute the workload associated with an Abaqus job between say 4 CPU cores on the same worker node we needa batch job submission script my_4_core_job.sge like the following:

#!/bin/bash#$ -cwd#$ -l rmem=8G#$ -pe openmp 4

module load apps/abaqus/2017/binary

abaqus job=my_job input=s4d.inp mp_mode=threads cpus=$NSLOTS scratch=$TMPDIR memory=→˓"32gb" interactive

Again, submit the job with:

qsub my_4_core_job.sge

Important notes:

• We specify the number of CPU cores using -pe openmp 4 near the top of the script. We tell Abaqus todistribute the work using cpus=$NSLOTS where NSLOTS is a variable automatically set by the job schedulerto be the same as the number at the end of the -pe openmp line.

• Here we request a job with 8GB of real memory per CPU core (-l rmem=8G) but Abaqus itself needs to betold the total amount of memory available (memory="32gb")

• The notes for the previous single-core batch job example still apply.

Using /fastdata as your Abaqus working directory

If you want to run Abaqus from a directory on /fastdata then you need to have the following line in your batch jobsubmission script just before the main abaqus command:

export BAS_DISABLE_FILE_LOCKING=1

Otherwise your Abaqus job will fail and you will see errors like the following in your my_job_name.dat outputfile:

***ERROR: An error occurred during a write access to<rank=0,arg_name=outdir>my_user_job.stt file. Check the disk spaceon your system.

This is a lie; Abaqus is failing to write the .stt file as it tries to use file locking which is not enabled on the/fastdata filesystem at present for performance reasons. Setting the BAS_DISABLE_FILE_LOCKING environ-ment variable to 1 is a Dassault Systems-approved workaround for this.

AdmixTools

2.2. Iceberg 53

https://en.wikipedia.org/wiki/File_locking


AdmixTools

Latest version 4.1

Dependancies Intel compiler 15.0.3

URL https://github.com/DReichLab/AdmixTools

AdmixTools implements five genetics methods described for learning about population mixtures. Details ofthese methods/algorithms can be found in Patterson et al. (2012) Ancient Admixture in Human History(doi:10.1534/genetics.112.145037).

Usage

Load version 4.1 of AdmixTools with the command

module load apps/intel/15/AdmixTools/4.1

This module makes the following programs available (by adding their containing directory to the PATH environmentvariable); programs; these are documented in README files contained in the $ADMIXTOOLSDIR directory.

• convertf: See README.CONVERTF for documentation of programs for converting file formats.

• qp3Pop: See README.3PopTest for details of running f_3 test. This test can be used as a format test ofadmixture with 3 populations.

• qpBound: See README.3PopTest for details of running qpBound. This test can be used for estimatingbounds on the admixture proportions, given 3 populations (2 reference and one target).

• qpDstat: See README.Dstatistics for details of running D-statistics. This is a formal test of admixturewith 4 populations.

• qpF4Ratio: See README.F4RatioTest for details of running F4 ratio estimation. This program computesthe admixture proportion by taking the ratio of two f4 tests.

• rolloff: See README.ROLLOFF/ README.ROLLOFF_OUTPUT for details for running rolloff. This pro-gram can be used for dating admixture events.

AdmixTools also comes with several examples. The example inputs and outputs can be found in $ADMIXTOOLSDIR/examples, whilst the example data the inputs reference are contained in $ADMIXTOOLSDIR/data

Installation Notes

Version 4.1

This script (and Makefile):

1. Checks out a specific git commit of AdmixTools (def3c5d75d1b10fd3270631f5c64adbf3af04d4d; NB the repos-itory does not contain any tags)

2. Builds GNU Scientific library (GSL) v2.2 and installs into the AdmixTools final installation directory (/usr/local/packages6/apps/intel/15/AdmixTools/4.1)

3. Builds AdmixTools (using v15.0.3 of the Intel Compiler and a customised Makefile (links against Intellibifcore rather than gfortran and uses MKL for BLAS and LAPACK functions)

4. Installs AdmixTools in the aforementioned directory.


https://github.com/DReichLab/AdmixTools

http://dx.doi.org/10.1534/genetics.112.145037i


5. Downloads the data needed to run the examples then runs the examples

6. Installs this module file as /usr/local/modulefiles/apps/intel/15/AdmixTools/4.1

Annovar

Annovar

Version 2015DEC14

URL http://annovar.openbioinformatics.org/en/latest/

ANNOVAR is an efficient software tool to utilize update-to-date information to functionally annotate genetic variantsdetected from diverse genomes (including human genome hg18, hg19, hg38, as well as mouse, worm, fly, yeast andmany others).

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand.

To add the annovar binaries to the system PATH, execute the following command

module load apps/binapps/annovar/2015DEC14

Documentation

The annovar manual is available online at http://annovar.openbioinformatics.org/en/latest/

Installation notes

The install is a collection of executable Perl scripts. Installation involves copying the directory and adding it to thePATH. It seems that annovar uses dates to distinguish between releases rather than version numbers

tar -xvzf ./annovar.latest.tar.gzmkdir -p mkdir -p /usr/local/packages6/apps/binapps/annovarmv annovar /usr/local/packages6/apps/binapps/annovar/2015DEC14

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/annovar/2015DEC14

Its contents are

#%Module1.0######################################################################### annovar modulefile##

2.2. Iceberg 55

https://genetics.med.harvard.edu/reich/Reich_Lab/Software_files/AdmixTools_Example_Data.tar.gz

http://annovar.openbioinformatics.org/en/latest/

http://annovar.openbioinformatics.org/en/latest/



proc ModulesHelp { } {puts stderr "Makes annovar 2015DEC14 available"

}

set ANNOVAR_DIR /usr/local/packages6/apps/binapps/annovar/2015Dec14

module-whatis "Makes annovar 2015DEC14 available"

prepend-path PATH $ANNOVAR_DIR

ANSYS

ANSYS

Version 14, 14.5, 15, 16.1, 17.2

Support Level FULL

Dependencies If using the User Defined Functions (UDF) will also need the following: For ANSYSMechanical, Workbench, CFX and AutoDYN : Intel 14.0 or above Compiler For Fluent : GCC4.6.1 or above

URL http://www.ansys.com/en_uk

Local URL http://www.shef.ac.uk/cics/research/software/fluent

The Ansys suite of programs can be used to numerically simulate a large variety of structural and fluid dynamicsproblems found in many engineering, physics, medical, aeronotics and automative industry applications.

Interactive usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive sesssion with the qsh command.Alternatively, if you require more memory, for example 16 GB, use the command qsh -l rmem=16G.

To make the default version of ANSYS available (currently version 16.1), run the following:

module load apps/ansys

Alternatively, you can make a specific version available with one of the following commands:

module load apps/ansys/17.2module load apps/ansys/16.1module load apps/ansys/15.0module load apps/ansys/14.5module load apps/ansys/14

You can then issue one of the following commands to run an ANSYS product. The teaching versions mentioned beloware installed for use during ANSYS and Fluent teaching labs and will only allow models of up to 500,000 elements.


http://www.ansys.com/en_uk

http://www.shef.ac.uk/cics/research/software/fluent


ansyswb : to run ANSYS workbenchansys : to run ANSYS Mechanical outside the workbenchansystext: to run line-mode version of ansysAnsys and Ansystext : to run the teaching license version of the above two commands.fluent : to run Fluent outside the workbenchfluentext: to run the line-mode version of FluentFluent and Fluentext : to run the teaching license version of the above two commands.icemcfx or icem: to run icemcfd outside the workbench.lsdyna : to run LS Dyna command line outside of the workbench (note: version→˓required i.e. lsdyna161)

Running Batch fluent and ansys jobs

The easiest way of running batch jobs for a particular version of ANSYS (e.g. 17.2) is:

module load apps/ansys/17.2runansys

Or the same for Fluent:

module load apps/ansys/17.2runfluent

The runfluent and runansys commands submit a Fluent journal or ANSYS input file into the batch system andcan take a number of different parameters, according to your requirements.

runfluent command

Just typing runfluent will display information on how to use it:

$ runfluentTHIS COMMAND SUBMITS SERIAL or PARALLEL FLUENT JOBS TO THE SUN GRID ENGINE--------------------------------------------------------------------------Usage: runfluent [2d,2ddp,3d or 3ddp] fluent_journal_file -time hh:mm:ss [-mem=nn]

[-rmem=nn] [-mail your_email_address] [-nq] [-parallel nprocs][optional_extra_→˓fluent_params]Where;All but the first two parameters are optional.

First parameter is the dimensionality of the problem.Second parameter, fluent_journal_file, is the file containing the fluent commands.Other 'optional' parameters are:

-time hh:mm:ss is the cpu time needed in hours:minutes:seconds-mem=nn is the virtual memory needed (Default=8G). Example: -mem 12G (for 12

→˓GBytes)-rmem=nn is the real memory needed.(Default=2G). Example: -rmem 4G (for 4 GBytes)-mail email_address. You will receive emails about the progress of your job

Example: -mail [email protected] is an optional parameter to submit without confirming-parallel nprocs : Only needed for parallel jobs to specify the no.of processors.-project project_name : The job will use a project allocation.fluent_params : any parameter not recognised will also be passed onto

the fluent startup script.

Example: runfluent 3d nozzle.jou -time 00:30:00 -mem=10GFluent journal files are essentially a sequence of Fluent Commands

2.2. Iceberg 57


you would have entered by starting fluent in non-gui modeHere is an example journal file:

/file/read-case test.cas/file/read-data test.dat/solve iter 200

/file/write-data testv5b.datyes

/exityes

Note that there can be no graphics output related commandsin the journal file as the job will be run in batch mode

Please see fluent documents for further details of journal files andhow to create them by typing- docs

You can startup an interactive fluent session in non-gui mode toexperiment. For example, by using the command: qrsh fluent 3d -g

Note that the option mem has been deprecated and is no longer required

An example of its usage:

runfluent 3d nozzle.jou -time 00:30:00 -rmem=10G

Fluent journal files are essentially a sequence of Fluent Commands you would have entered by starting fluent innon-GUI mode.

Here is an example journal file:

/file/read-case test.cas/file/read-data test.dat/solve iter 200/file/write-data testv5b.datyes/exityes

Note that there can be no graphics-output-related commands in the journal file as the job will be run in batch (non-interative) mode. Please see the Fluent documentation for further details of journal files and how to create them.

By using the -g parameter, you can startup an interactive Fluent session in non-GUI mode to experiment. For example:

fluent 3d -g

runansys command

Just typing runansys will display information on how to use it:

$ runansys

**Input ( .dat or .inp) file containing Ansys commands was not specified.

THIS COMMAND SUBMITS ANSYS JOBS TO THE SUN GRID ENGINE------------------------------------------------------Usage: runansys ansys_inp_file [-time hh:mm:ss][-mem=nn] [-rmem=nn] [-parallel n]

[-usefastdata] [-project proj_name] [-mail email_address] [other qsub→˓parameters]

Where;ansys_inp_file is a file containing a series of Ansys commands.



-time hh:mm:ss is the cpu time needed in hours:minutes:seconds,if not specified 1 hour will be assumed.

-mem=nn is the virtual memory requirement.-rmem=nn is the real memory requirement.-parallel n request an n-way parallel ansys job-gpu use GPU

Note for GPU users: -mem= must be greater than 18G.-usefastdata Use /fastdata/te1st as the working directory for temporary files-project project_name : The job will use a project's allocation.-mail your_email_address : Job progress report is emailed to you.

As well as time and memory, any other valid qsub parameter can be specified.

All parameters except the ansys_inp file are optional.

Output files created by Ansys take their names fromthe jobname specified by the user.

You will be prompted for a jobname as well as any otherstartup parameter you wish to pass to Ansys

Example:runansys test1.dat -time 00:30:00 -mem 8G -rmem=3G -mail [email protected]

Note that the option mem has been deprecated and is no longer required.

bcbio

bcbio

Latest version 0.9.6a

Dependancies gcc 5.2, R 3.2.1, Anaconda Python 2.3

URL http://bcbio-nextgen.readthedocs.org/en/latest/

A python toolkit providing best-practice pipelines for fully automated high throughput sequencing analysis. You writea high level configuration file specifying your inputs and analysis parameters. This input drives a parallel pipeline thathandles distributed execution, idempotent processing restarts and safe transactional steps. The goal is to provide ashared community resource that handles the data processing component of sequencing analysis, providing researcherswith more time to focus on the downstream biology.

Usage

Load version 0.9.6a of bcbio with the command

module load apps/gcc/5.2/bcbio/0.9.6a

There is also a development version of bcbio installed on iceberg. This could change without warning and should notbe used for production

module load apps/gcc/5.2/bcbio/devel

These module commands add bcbio commands to the PATH, load any supporting environments and correctly configurethe system for bcbio usage.

2.2. Iceberg 59

http://bcbio-nextgen.readthedocs.org/en/latest/


Once the module is loaded you can, for example, check the version of bcbio

bcbio_nextgen.py -v

/usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/anaconda/lib/python2.7/site-packages/→˓matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and→˓replaced with axes.prop_cycle; please use the latter.warnings.warn(self.msg_depr % (key, alt_key))

0.9.6a

To check how the loaded version of bcbio has been configured

more $BCBIO_DIR/config/install-params.yaml

At the time of writing, the output from the above command is

aligners:- bwa- bowtie2- rtg- hisat2genomes:- hg38- hg19- GRCh37isolate: truetooldir: /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/toolstoolplus: []

Example batch submission

TODO

Integration with SGE

TODO

Installation Notes


0.9.6a

Version 0.9.6a was installed using gcc 5.2, R 3.2.1 and Anaconda Python 2.3. The install was performed in two parts.

The first step was to run the SGE script below in batch mode. Note that the install often fails due to external servicesbeing flaky. See https://github.com/rcgsheffield/sheffield_hpc/issues/219 for details. Depending on the reason for thefailure, it should be OK to simply restart the install. This particular install was done in one-shot...no restarts necessary.

• install_bcbio_0.96a

The output from this batch run can be found in /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/install_output/

Once the install completed, the module file (see Modulefile section) was created and loaded and the following upgradeswere performed


https://github.com/rcgsheffield/sheffield_hpc/issues/219


bcbio_nextgen.py upgrade --toolplus gatk=./GenomeAnalysisTK.jarbcbio_nextgen.py upgrade --genomes hg38 --aligners hisat2

The GATK .jar file was obtained from https://www.broadinstitute.org/gatk/download/

A further upgrade was performed on 13th January 2016. STAR had to be run directly because the bcbio upgradecommand that made use of it kept stalling ( bcbio_nextgen.py upgrade –data –genomes GRCh37 –aligners bwa –aligners star ). We have no idea why this made a difference but at least the direct STAR run could make use ofmultiple cores whereas the bcbio installer only uses 1

#!/bin/bash#$ -l rmem=3G#$ -P radiant#$ -pe openmp 16

module load apps/gcc/5.2/bcbio/0.9.6aSTAR --genomeDir /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/genomes/Hsapiens/→˓GRCh37/star --genomeFastaFiles /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/→˓genomes/Hsapiens/GRCh37/seq/GRCh37.fa --runThreadN 16 --runMode genomeGenerate --→˓genomeSAindexNbases 14

bcbio_nextgen.py upgrade --data --genomes GRCh37 --aligners bwa

Another upgrade was performed on 25th February 2016

module load apps/gcc/5.2/bcbio/0.9.6abcbio_nextgen.py upgrade -u stable --data --genomes mm10 --aligners star --aligners→˓bwa

As is usually the case for us, this stalled on the final STAR command. The exact call to STAR was found in/usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/genomes/Mmusculus/mm10/star/Log.out and run manually in a 16core OpenMP script:

STAR --runMode genomeGenerate --runThreadN 16 --genomeDir /usr/local/packages6/→˓apps/gcc/5.2/bcbio/0.9.6a/genomes/Mmusculus/mm10/star --genomeFastaFiles /usr/→˓local/packages6/apps/gcc/5.2/bcbio/0.9.6a/genomes/Mmusculus/mm10/seq/mm10.fa --→˓genomeSAindexNbases 14 --genomeChrBinNbits 14

This failed (see https://github.com/rcgsheffield/sheffield_hpc/issues/272). The fix was to add the line

index mm10 /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/genomes/Mmusculus/mm10/seq/→˓mm10.fa

to the file

usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/galaxy/tool-data/sam_fa_indices.loc

Update: 14th March 2016

Another issue required us to modify /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/genomes/Mmusculus/mm10/seq/mm10-resources.yaml so that it read

version: 16

aliases:snpeff: GRCm38.82ensembl: mus_musculus_vep_83_GRCm38

2.2. Iceberg 61

https://www.broadinstitute.org/gatk/download/




variation:dbsnp: ../variation/mm10-dbSNP-2013-09-12.vcf.gzlcr: ../coverage/problem_regions/repeats/LCR.bed.gz

rnaseq:transcripts: ../rnaseq/ref-transcripts.gtftranscripts_mask: ../rnaseq/ref-transcripts-mask.gtftranscriptome_index:tophat: ../rnaseq/tophat/mm10_transcriptome.ver

dexseq: ../rnaseq/ref-transcripts.dexseq.gff3refflat: ../rnaseq/ref-transcripts.refFlatrRNA_fa: ../rnaseq/rRNA.fa

srnaseq:srna-transcripts: ../srnaseq/srna-transcripts.gtfmirbase-hairpin: ../srnaseq/hairpin.famirbase-mature: ../srnaseq/hairpin.famirdeep2-fasta: ../srnaseq/Rfam_for_miRDeep.fa

Development version

The development version was installed using gcc 5.2, R 3.2.1 and Anaconda Python 2.3.

• install_bcbio_devel.sge This is a SGE submit script. The long running time of the installer made itbetter-suited to being run as a batch job.

• bcbio-devel modulefile located on the system at /usr/local/modulefiles/apps/gcc/5.2/bcbio/devel

The first install attempt failed with the error

To debug, please try re-running the install command with verbose output:export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export SHELL=$→˓{SHELL:-/bin/bash} && export PERL5LIB=/usr/local/packages6/apps/gcc/5.2/bcbio/devel/→˓tools/lib/perl5:${PERL5LIB} && /usr/local/packages6/apps/gcc/5.2/bcbio/devel/tools/→˓bin/brew install -v --env=inherit --ignore-dependencies gitTraceback (most recent call last):

File "bcbio_nextgen_install.py", line 276, in <module>main(parser.parse_args(), sys.argv[1:])

File "bcbio_nextgen_install.py", line 46, in mainsubprocess.check_call([bcbio["bcbio_nextgen.py"], "upgrade"] + _clean_args(sys_

→˓argv, args, bcbio))File "/usr/local/packages6/apps/binapps/anacondapython/2.3/lib/python2.7/subprocess.

→˓py", line 540, in check_callraise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['/usr/local/packages6/apps/gcc/5.2/bcbio/→˓devel/anaconda/bin/bcbio_nextgen.py', 'upgrade', '--tooldir=/usr/local/packages6/→˓apps/gcc/5.2/bcbio/devel/tools', '--isolate', '--genomes', 'GRCh37', '--aligners',→˓'bwa', '--aligners', 'bowtie2', '--data']' returned non-zero exit status 1

I manually ran the command

export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export SHELL=$→˓{SHELL:-/bin/bash} && export PERL5LIB=/usr/local/packages6/apps/gcc/5.2/bcbio/devel/→˓tools/lib/perl5:${PERL5LIB} && /usr/local/packages6/apps/gcc/5.2/bcbio/devel/tools/→˓bin/brew install -v --env=inherit --ignore-dependencies git

and it completed successfully. I then resubmitted the submit script which eventually completed successfully. It tookseveral hours! At this point, I created the module file.



Bcbio was upgraded to the development version with the following interactive commands

module load apps/gcc/5.2/bcbio/develbcbio_nextgen.py upgrade -u development

The GATK .jar file was obtained from https://www.broadinstitute.org/gatk/download/ and installed to bcbio by runningthe following commands interactively

module load apps/gcc/5.2/bcbio/develbcbio_nextgen.py upgrade --tools --toolplus gatk=./cooper/GenomeAnalysisTK.jar

Module files

• 0.9.6a

Testing

Version 0.9.6a

The following test script was submitted to the system as an SGE batch script

#!/bin/bash#$ -pe openmp 12#$ -l rmem=4G #Per Core!

module add apps/gcc/5.2/bcbio/0.9.6a

git clone https://github.com/chapmanb/bcbio-nextgen.gitcd bcbio-nextgen/tests./run_tests.sh devel./run_tests.sh rnaseq

The tests failed due to a lack of pandoc

[2016-01-07T09:40Z] Error: pandoc version 1.12.3 or higher is required and was not→˓found.[2016-01-07T09:40Z] Execution halted[2016-01-07T09:40Z] Skipping generation of coverage report: Command 'set -o pipefail;→˓/usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/anaconda/bin/Rscript /data/te1st/→˓bcbio-nextgen/tests/test_automated_output/report/qc-coverage-report-run.RError: pandoc version 1.12.3 or higher is required and was not found.Execution halted' returned non-zero exit status 1

The full output of this testrun is on the system at /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/tests/7-jan-2016/

Pandoc has been added to the list of applications that need to be installed on iceberg.

Development version

The following test script was submitted to the system. All tests passed. The output is at /usr/local/packages6/apps/gcc/5.2/bcbio/0.9.6a/tests/tests_07_01_2016/

#!/bin/bash#$ -pe openmp 12

2.2. Iceberg 63

https://www.broadinstitute.org/gatk/download/


#$ -l rmem=4G #Per Core!

module add apps/gcc/5.2/bcbio/0.9.6a

git clone https://github.com/chapmanb/bcbio-nextgen.gitcd bcbio-nextgen/tests./run_tests.sh devel./run_tests.sh rnaseq

bcl2fastq

bcl2fastq

Versions 1.8.4

Support Level Bronze

URL http://support.illumina.com/downloads/bcl2fastq_conversion_software_184.html

Illumina sequencing instruments generate per-cycle BCL basecall files as primary sequencing output, but many down-stream analysis applications use per-read FASTQ files as input. bcl2fastq combines these per-cycle BCL files froma run and translates them into FASTQ files. bcl2fastq can begin bcl conversion as soon as the first read has beencompletely sequenced.

Usage

To make bcl2fastq available, use the following module command in your submission scripts

module load apps/bcl2fastq/1.8.4

Installation Notes

These notes are primarily for system administrators.

Compilation was done using gcc 4.4.7. I tried it with gcc 4.8 but ended up with a lot of errors. The package is alsodependent on Perl. Perl 5.10.1 was used which was the system Perl installed at the time. The RPM Perl-XML-Simplealso needed installing.

export TMP=/tmpexport SOURCE=${TMP}/bcl2fastqexport BUILD=${TMP}/bcl2fastq-1.8.4-buildmkdir -p /usr/local/packages6/apps/gcc/4.4.7/bcl2fastq/1.8.4export INSTALL=/usr/local/packages6/apps/gcc/4.4.7/bcl2fastq/1.8.4

mv bcl2fastq-1.8.4.tar.bz2 ${TMP}cd ${TMP}tar xjf bcl2fastq-1.8.4.tar.bz2

mkdir ${BUILD}cd ${BUILD}${SOURCE}/src/configure --prefix=${INSTALL}


http://support.illumina.com/downloads/bcl2fastq_conversion_software_184.html


makemake install

bedtools

bedtools

Versions 2.25.0

Dependancies compilers/gcc/5.2

URL https://bedtools.readthedocs.org/en/latest/

Collectively, the bedtools utilities are a swiss-army knife of tools for a wide-range of genomics analysis tasks.

Interactive usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh command.

The latest version of bedtools (currently version 2.25.0) is made available with the command:

module load apps/gcc/5.2/bedtools

Alternatively, you can make a specific version available:

module load apps/gcc/5.2/bedtools/2.25.0

After that any of the bedtools commands can be run from the prompt.

Installation notes

bedtools was installed using gcc 5.2 with the script install_bedtools.sh

Testing


Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/bedtools/2.25.0

The contents of the module file is

#%Module1.0######################################################################### bedtools module file##

## Module file logging

2.2. Iceberg 65

https://bedtools.readthedocs.org/en/latest/


source /usr/local/etc/module_logging.tcl##

proc ModulesHelp { } {global bedtools-version

puts stderr " Adds `bedtools-$bedtools-version' to your PATH environment→˓variable and necessary libraries"}

set bedtools-version 2.25.0module load compilers/gcc/5.2

prepend-path PATH /usr/local/packages6/apps/gcc/5.2/bedtools/2.25.0/bin

bitseq

bitseq

Versions 0.7.5

URL http://bitseq.github.io/

Transcript isoform level expression and differential expression estimation for RNA-seq

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrsh command

module load apps/gcc/5.2/bitseq


module load apps/gcc/5.2/bitseq/0.7.5

This command adds the BitSeq binaries to your PATH.

Documentation

Documentation is available online http://bitseq.github.io/howto/index

Installation notes

BitSeq was installed using gcc 5.2

module load compilers/gcc/5.2tar -xvzf ./BitSeq-0.7.5.tar.gzcd BitSeq-0.7.5

make


http://bitseq.github.io/

http://bitseq.github.io/howto/index


cd ..mkdir -p /usr/local/packages6/apps/gcc/5.2/bitseqmv ./BitSeq-0.7.5 /usr/local/packages6/apps/gcc/5.2/bitseq/

Testing


Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/bitseq/0.7.5. It’s contents

#%Module1.0######################################################################### bitseq 0.7.5 modulefile##



proc ModulesHelp { } {puts stderr "Makes bitseq 0.7.5 available"

}

set version 0.7.5set BITSEQ_DIR /usr/local/packages6/apps/gcc/5.2/bitseq/BitSeq-$version

module-whatis "Makes bitseq 0.7.5 available"

prepend-path PATH $BITSEQ_DIR

BLAST

ncbi-blast

Version 2.3.0

URL https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=Download

BLAST+ is a new suite of BLAST tools that utilizes the NCBI C++ Toolkit. The BLAST+ applications have a numberof performance and feature improvements over the legacy BLAST applications. For details, please see the BLAST+user manual.

2.2. Iceberg 67

https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=Download

https://www.ncbi.nlm.nih.gov/books/NBK279690/

https://www.ncbi.nlm.nih.gov/books/NBK279690/


Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx or qrshcommand.

The latest version of BLAST+ (currently 2.3.0) is made available with the command

module load apps/binapps/ncbi-blast


module load apps/binapps/ncbi-blast/2.3.0

This command makes the BLAST+ executables available to your session by adding the install directory to your PATHvariable. It also sets the BLASTDB database environment variable.

You can now run commands directly. e.g.

blastn -help

Databases

The following databases have been installed following a user request

• nr.*tar.gz Non-redundant protein sequences from GenPept, Swissprot, PIR, PDF, PDB, and NCBI RefSeq

• nt.*tar.gz Partially non-redundant nucleotide sequences from all traditional divisions of GenBank, EMBL, andDDBJ excluding GSS,STS, PAT, EST, HTG, and WGS.

A full list of databases available on the NCBI FTP site is at https://ftp.ncbi.nlm.nih.gov/blast/documents/blastdb.html

If you need any of these installing, please make a request on our github issues log.

Installation notes

This was an install from binaries

#get binaries and put in the correct locationwget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/LATEST/ncbi-blast-2.3.0+-x64-linux.→˓tar.gztar -xzf ./ncbi-blast-2.3.0+-x64-linux.tar.gzmkdir -p /usr/local/packages6/apps/binapps/ncbi-blast/mv ncbi-blast-2.3.0+ /usr/local/packages6/apps/binapps/ncbi-blast/

#Create database directorymkdir -p /usr/local/packages6/apps/binapps/ncbi-blast/ncbi-blast-2.3.0+/db

#Install the nr databasecd /usr/local/packages6/apps/binapps/ncbi-blast/ncbi-blast-2.3.0+/dbfor num in `seq 0 48`;dopaddednum=`printf "%02d" $num``wget ftp://ftp.ncbi.nlm.nih.gov/blast/db/nr.$paddednum.tar.gz`done

#Install the nt database


https://ftp.ncbi.nlm.nih.gov/blast/documents/blastdb.html


for num in `seq 0 36`;dopaddednum=`printf "%02d" $num``wget ftp://ftp.ncbi.nlm.nih.gov/blast/db/nt.$paddednum.tar.gz`done

for f in *.tar.gz; do tar -xvzf $f; done

Testing

No testing has been performed. If you can suggest a suitable test suite, please contact us.

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/ncbi-blast/2.3.0


#%Module1.0######################################################################### BLAST 2.3.0 modulefile##


proc ModulesHelp { } {puts stderr "Makes BLAST 2.3.0 available"

}

set BLAST_DIR /usr/local/packages6/apps/binapps/ncbi-blast/ncbi-blast-2.3.0+

module-whatis "Makes BLAST 2.3.0 available"

prepend-path PATH $BLAST_DIR/binprepend-path BLASTDB $BLAST_DIR/db

bless

BLESS

Version 1.02

URL http://sourceforge.net/p/bless-ec/wiki/Home/

BLESS: Bloom-filter-based Error Correction Solution for High-throughput Sequencing Reads

2.2. Iceberg 69

http://sourceforge.net/p/bless-ec/wiki/Home/


Interactive Usage

BLESS uses MPI and we currently have no interactive MPI environments available. As such, it is not possible to runBLESS interactively on Iceberg.

Batch Usage

The latest version of bless (currently 1.02) is made available with the command

module load apps/gcc/4.9.2/bless/1.02

The module create the environment variable BLESS_PATH which points to the BLESS installation directory. It alsoloads the dependent modules

• compilers/gcc/4.9.2

• mpi/gcc/openmpi/1.8.3

Here is an example batch submission script that makes use of two example input files we have included in our instal-lation of BLESS:

1 #!/bin/bash2 # Next line is memory per slot3 #$ -l rmem=5G4 # Ask for 4 slots in an OpenMP/MPI Hybrid queue5 # These 4 slots will all be on the same node6 #$ -pe openmpi-hybrid-4 47

8 # load the module9 module load apps/gcc/4.9.2/bless/1.02

10

11 # Output information about nodes where code is running12 cat $PE_HOSTFILE > nodes13

14 data_folder=$BLESS_PATH/test_data15 file1=test_small_1.fq16 file2=test_small_2.fq17

18 # Number of cores per node we are going to use.19 cores=420 export OMP_NUM_THREADS=$cores21

22 # Run BLESS23 mpirun $BLESS_PATH/bless -kmerlength 21 -smpthread $cores -prefix test_bless -read1

→˓$data_folder/$file1 -read2 $data_folder/$file2

BLESS makes use of both MPI and OpenMP parallelisation frameworks. As such, it is necessary to use a HybridMPI/SMP Parallel environment. The current build of BLESS does not work on more than one node. This will limityou to the maximum number of cores available on one node.

For example, to use all 16 cores on a 16 core node you would request the following parallel environment

#$ -pe openmpi-hybrid-16 16

Remember that memory is allocated on a per-slot basis. You should ensure that you do not request more memory thanis available on a single node or your job will be permanently stuck in a queue-waiting (qw) status.



Installation notes

Various issues were encountered while attempting to install BLESS. It was necessary to install gcc 4.9.2 in order tobuild BLESS. No other compiler worked!

Here are the install steps

tar -xvzf ./bless.v1p02.tgz

mkdir -p /usr/local/modulefiles/apps/gcc/4.9.2/bless/cd v1p02/

Load Modules

module load compilers/gcc/4.9.2module load mpi/gcc/openmpi/1.8.3

Modify the Makefile. Change the line

cd zlib; ./compile

to

cd zlib;

Manually compile zlib

cd zlib/./compile

Finish the compilation

cd ..make

Copy the bless folder to the central location

cd ..cp -r ./v1p02/ /usr/local/packages6/apps/gcc/4.9.2/bless/

Testing


Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/4.9.2/bless/1.02

• The module file is on github.

Bowtie2

2.2. Iceberg 71



bowtie2

Versions 2.2.26

URL http://bowtie-bio.sourceforge.net/bowtie2/index.shtml

Bowtie 2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. It isparticularly good at aligning reads of about 50 up to 100s or 1,000s of characters, and particularly good at aligningto relatively long (e.g. mammalian) genomes. Bowtie 2 indexes the genome with an FM Index to keep its memoryfootprint small: for the human genome, its memory footprint is typically around 3.2 GB. Bowtie 2 supports gapped,local, and paired-end alignment modes.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive sesssion with the qsh or qrshcommand.

The latest version of bowtie2 (currently 2.2.26) is made available with the command

module load apps/gcc/5.2/bowtie2


module load apps/gcc/5.2/bowtie2/2.2.6

This command makes the bowtie2 executables available to your session by adding the install directory to your PATHvariable. This allows you to simply do something like the following

bowtie2 --version

which gives results that looks something like

/usr/local/packages6/apps/gcc/5.2/bowtie2/2.2.6/bowtie2-align-s version 2.2.664-bitBuilt on node063Fri Oct 23 08:40:38 BST 2015Compiler: gcc version 5.2.0 (GCC)Options: -O3 -m64 -msse2 -funroll-loops -g3 -DPOPCNT_CAPABILITYSizeof {int, long, long long, void*, size_t, off_t}: {4, 8, 8, 8, 8, 8}

Installation notes

bowtie2 2.2.6 was installed using gcc 5.2

#Buildmodule load compilers/gcc/5.2unzip bowtie2-2.2.6-source.zipcd bowtie2-2.2.6make

#Installcd ..mkdir -p /usr/local/packages6/apps/gcc/5.2/bowtie2mv ./bowtie2-2.2.6 /usr/local/packages6/apps/gcc/5.2/bowtie2/2.2.6


http://bowtie-bio.sourceforge.net/bowtie2/index.shtml


Testing


Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/bowtie2/2.2.6


#%Module1.0######################################################################### bowtie2 2.2.6 modulefile##



proc ModulesHelp { } {puts stderr "Makes bowtie 2.2.6 available"

}

set version 2.2.6set BOWTIE2_DIR /usr/local/packages6/apps/gcc/5.2/bowtie2/$version

module-whatis "Makes bowtie2 v2.2.6 available"

prepend-path PATH $BOWTIE2_DIR

bwa

bwa

Versions 0.7.12

URL http://bio-bwa.sourceforge.net/

BWA (Burrows-Wheeler Aligner) is a software package for mapping low-divergent sequences against a large referencegenome, such as the human genome. It consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM. Thefirst algorithm is designed for Illumina sequence reads up to 100bp, while the rest two for longer sequences rangedfrom 70bp to 1Mbp. BWA-MEM and BWA-SW share similar features such as long-read support and split alignment,but BWA-MEM, which is the latest, is generally recommended for high-quality queries as it is faster and more accurate.BWA-MEM also has better performance than BWA-backtrack for 70-100bp Illumina reads.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx command.

2.2. Iceberg 73

http://bio-bwa.sourceforge.net/


The latest version of bwa (currently 0.7.12) is made available with the command

module load apps/gcc/5.2/bwa


module load apps/gcc/5.2/bwa/0.7.12module load apps/gcc/5.2/bwa/0.7.5a

This command makes the bwa binary available to your session.

Documentation

Once you have made bwa available to the system using the module command above, you can read the man pages bytyping

man bwa

Installation notes

bwa 0.7.12

bwa 0.7.12 was installed using gcc 5.2


#buildmodule load compilers/gcc/5.2tar -xvjf ./bwa-0.7.12.tar.bz2cd bwa-0.7.12make

#Sort out manfilemkdir -p share/man/man1mv bwa.1 ./share/man/man1/

#Installmkdir -p /usr/local/packages6/apps/gcc/5.2/bwa/cd ..mv bwa-0.7.12 /usr/local/packages6/apps/gcc/5.2/bwa/0.7.12/

bwa 0.7.5a

bwa 0.7.5a was installed using gcc 5.2


#buildmodule load compilers/gcc/5.2tar -xvjf bwa-0.7.5a.tar.bz2cd bwa-0.7.5amake

#Sort out manfilemkdir -p share/man/man1mv bwa.1 ./share/man/man1/



#Installmkdir -p /usr/local/packages6/apps/gcc/5.2/bwa/cd ..mv bwa-0.7.5a /usr/local/packages6/apps/gcc/5.2/bwa/

Testing


Module files

The default version is controlled by the .version file at /usr/local/modulefiles/apps/gcc/5.2/bwa/.version

#%Module1.0######################################################################### version file for bwa##set ModulesVersion "0.7.12"

Version 0.7.12

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/bwa/0.7.12

• On github: 0.7.12.

Version 0.7.5a

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/bwa/0.7.5a

• On github: 0.7.5a.

CDO

CDO

Version 1.7.2

URL https://code.zmaw.de/projects/cdo

CDO is a collection of command line Operators to manipulate and analyse Climate and NWP model Data. Supporteddata formats are GRIB 1/2, netCDF 3/4, SERVICE, EXTRA and IEG. There are more than 600 operators available.

Interactive Usage


To add the cdo command to the system PATH, execute the following command

module load apps/gcc/5.3/cdo/1.7.2

2.2. Iceberg 75

https://code.zmaw.de/projects/cdo


Documentation

The CDO manual is available online at https://code.zmaw.de/projects/cdo/embedded/index.html

Installation notes

Installation

(1) Download the latest current version:

wget https://code.zmaw.de/attachments/download/12350/cdo-current.tar.gz

(2) Extract the files into a working directory ( I have used /data/te1st/→˓2016/cdo ) :

gunzip cdo-current.tar.gztar -xvf cdo-current.tar

(3) Install the program ( I have used /usr/local/extras/CDO ):

module load compilers/gcc/5.3./configure --prefix=/usr/local/extras/CDO --with-netcdf=/usr/local/

→˓extras/netcdf/4.3.2make install

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.3/cdo

Its contents are

#%Module1.0######################################################################### CDO Climate Data Operators Module file.##


proc ModulesHelp { } {puts stderr "Makes Climate Data Operator $version available"

}

set version 1.7.2set BIN_DIR /usr/local/extras/CDO/$version

module-whatis "Makes CDO (Climate Data Operators) $version available"

prepend-path PATH $BIN_DIR/bin

Code Saturne 4.0


https://code.zmaw.de/projects/cdo/embedded/index.html


Code Saturne

Version 4.0

URL http://code-saturne.org/cms/

Documentation http://code-saturne.org/cms/documentation

Location /usr/local/packages6/apps/gcc/4.4.7/code_saturne/4.0

Code_Saturne solves the Navier-Stokes equations for 2D, 2D-axisymmetric and 3D flows, steady or unsteady, laminaror turbulent, incompressible or weakly dilatable, isothermal or not, with scalars transport if required.

Usage

To make code saturne available, run the following module command after starting a qsh session.

module load apps/code_saturne/4.0.0

Troubleshooting

If you run Code Saturne jobs from /fastdata they will fail since the underlying filesystem used by /fastdatadoes not support posix locks. If you need more space than is available in your home directory, use /data instead. Ifyou use /fastdata with Code Saturne you may get an error message similar to the one below

File locking failed in ADIOI_Set_lock(fd 14,cmd F_SETLKW/7,type F_WRLCK/1,whence 0)→˓with return value FFFFFFFF and errno 26.- If the file system is NFS, you need to use NFS version 3, ensure that the lockd→˓daemon is running on all the machines, and mount the directory with the 'noac'→˓option (no attribute caching).- If the file system is LUSTRE, ensure that the directory is mounted with the 'flock'→˓option.ADIOI_Set_lock:: Function not implementedADIOI_Set_lock:offset 0, length 8--------------------------------------------------------------------------MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLDwith errorcode 1.NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.You may or may not see output from other processes, depending onexactly when Open MPI kills them.--------------------------------------------------------------------------solver script exited with status 1.

Error running the calculation.Check Code_Saturne log (listing) and error* files for details.

Installation Notes


Installation notes for the version referenced by module module load apps/code_saturne/4.0.0:

Pre-requisites: This version of Code Saturne was built with the following:-

• gcc 4.4.7

• cgns 3.2.1

2.2. Iceberg 77

http://code-saturne.org/cms/

http://code-saturne.org/cms/documentation


• MED 3.0.8

• OpenMPI 1.8.3

• HDF5 1.8.14 built using gcc

module load libs/gcc/4.4.7/cgns/3.2.1

tar -xvzf code_saturne-4.0.0.tar.gzmkdir code_saturn_buildcd code_saturn_build/./../code_saturne-4.0.0/configure --prefix=/usr/local/packages6/apps/gcc/4.4.7/code_→˓saturne/4.0 --with-mpi=/usr/local/mpi/gcc/openmpi/1.8.3/ --with-med=/usr/local/→˓packages6/libs/gcc/4.4.7/med/3.0.8/ --with-cgns=/usr/local/packages6/libs/gcc/4.4.7/→˓cgnslib/3.2.1 --with-hdf5=/usr/local/packages6/hdf5/gcc-4.4.7/openmpi-1.8.3/hdf5-1.→˓8.14/

This gave the following configuration

Configuration options:use debugging code: noMPI (Message Passing Interface) support: yesOpenMP support: no

The package has been configured. Type:makemake install

To generate and install the PLE package

Configuration options:use debugging code: nouse malloc hooks: nouse graphical user interface: yesuse long integers: yesZlib (gzipped file) support: yesMPI (Message Passing Interface) support: yesMPI I/O support: yesMPI2 one-sided communication support: yes

OpenMP support: noBLAS (Basic Linear Algebra Subprograms) support: noLibxml2 (XML Reader) support: yesParMETIS (Parallel Graph Partitioning) support: noMETIS (Graph Partitioning) support: noPT-SCOTCH (Parallel Graph Partitioning) support: noSCOTCH (Graph Partitioning) support: noCCM support: noHDF (Hierarchical Data Format) support: yesCGNS (CFD General Notation System) support: yesMED (Model for Exchange of Data) support: yesMED MPI I/O support: yes

MEDCoupling support: noCatalyst (ParaView co-processing) support: noEOS support: nofreesteam support: noSALOME GUI support: yesSALOME Kernel support: yesDynamic loader support (for YACS): dlopen



I then did

makemake install

Post Install Steps

To make Code Saturne aware of the SGE system:

• Created /usr/local/packages6/apps/gcc/4.4.7/code_saturne/4.0/etc/code_saturne.cfg: See code_saturne.cfg 4.0

• Modified /usr/local/packages6/apps/gcc/4.4.7/code_saturne/4.0/share/code_saturne/batch/batch.SGE. See: batch.SGE 4.0

Testing

This module has not been yet been properly tested and so should be considered experimental.

Several user’s jobs up to 8 cores have been submitted and ran to completion.

Module File

Module File Location: /usr/local/modulefiles/apps/code_saturne/4.0.0

#%Module1.0######################################################################### code_saturne 4.0 module file##


proc ModulesHelp { } {global code-saturneversion

puts stderr " Adds `code_saturn-$codesaturneversion' to your PATH environment→˓variable and necessary libraries"}

set codesaturneversion 4.0.module load mpi/gcc/openmpi/1.8.3

module-whatis "loads the necessary `code_saturne-$codesaturneversion' library paths"

set cspath /usr/local/packages6/apps/gcc/4.4.7/code_saturne/4.0prepend-path MANPATH $cspath/share/manprepend-path PATH $cspath/bin

COMSOL Multiphysics

2.2. Iceberg 79


COMSOL

Version 5.2

Support Level ?

Dependencies None

URL https://www.comsol.com/comsol-multiphysics

COMSOL Multiphysics provides a range of simulation, finite element analysis and solver functionality.

Availability, licensing and modules

COMSOL Multiphysics is installed on Iceberg but is not presently supported by CiCS. There is no site license forCOMSOL so to run it on Iceberg you will need either your own license or access to licenses managed by others. Forexample, a research group may operate a license server that can issue COMSOL licenses to (approved) users. If youneed access to a license on a temporary basis then you could try contacting Prof. Will Zimmerman, who has purchasedseveral licenses for his research group and may be willing to hire out licenses to other researchers. See Interactiveusage for information on how to specify a COMSOL license when starting COMSOL on Iceberg.

The following COMSOL Multiphysics 5.2 modules are installed:

• ACDC

• Acoustics

• Batteries and Fuel Cells

• CAD Import

• CFD

• Chemical Reaction Engineering

• COMSOL Multiphysics

• Corrosion

• Design

• ECAD Import

• Electrochemistry

• Electrodeposition

• Fatigue

• Geomechanics

• Heat Transfer

• LiveLink for Excel

• LiveLink for MATLAB

• MEMS

• Microfluidics

• Mixer

• Molecular Flow


https://www.comsol.com/comsol-multiphysics

https://www.sheffield.ac.uk/cbe/staff/staffprofiles/wzimmerman


• Multibody Dynamics

• Nonlinear Structural Materials

• Optimization

• Particle Tracing

• Pipe Flow

• Plasma

• Ray Optics

• RF

• Semiconductor

• Structural Mechanics

• Subsurface Flow

• Wave Optics

Interactive usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive graphical session with the qrshxcommand. Alternatively, if you require more memory, for example 16 GB, use the command qrshx -l rmem=16G

Next, run the following to make COMSOL Multiphysics available in your current session:

module load apps/binapps/comsol/5.2

Specify the location of your license information:

export LM_LICENSE_SERVER=/home/myusername/path/to/mylicensefile.dat:$LM_LICENSE_SERVER

Finally, start COMSOL Multiphysics:

comsol

The COMSOL Multiphysics user interface should then appear. Here mylicensefile.dat is a file containingeither:

• details of your license (which components you can use) or

• details of the license server that you want COMSOL to request licenses from.

If you are using a license server then your license file needs to contain no more than the following:

SERVER mylicenseserver.sheffield.ac.uk ANY 65321USE_SERVER

where mylicenseserver.sheffield.ac.uk is the hostname of your license server and 654321 is the portto connect to on that machine to request a COMSOL license.

The person responsible for managing the license server may ask for your Iceberg username to allow you to requestlicenses (whilst preventing others from doing so).

2.2. Iceberg 81


Serial batch usage

To submit a batch job that will run on just one CPU core create a file called e.g. myjob.sge:

#!/bin/bash#$ -l h_rt=02:00:00#$ -l rmem=16G#$ -m bea#$ -M [email protected]#$ -j y#$ -o myjob.log

module load apps/binapps/comsol/5.2export _JAVA_OPTIONS="-Xmx2g"

export LM_LICENSE_FILE=/path/to/mylicence.datINPFILE=myinput.mphOUTFILE=myoutput.mphBATCHLOG=myjob_comsol_own_log.log

comsol batch -tmpdir $TMPDIR -inputfile $INPFILE -outputfile $OUTFILE -batchlog→˓$BATCHLOG

You will need to change:

• The email address used for email notifications;

• The path to your COMSOL license file;

• The name of the COMSOL input file (with a .mph extension).

You may want to change:

• The amount of time that the job is allowed to run for;

• The amount of real (rmem) memory that the job can use;

• The maximum amount of virtual memory used by Java (-Xmx2G means that it can use 2GB of virtual memory);

• The name of the output file generated by COMSOL (has a .mph extension);

• The name of the BATCHLOG log file generated by COMSOL;

• The name of the log file generated by the cluster’s scheduling software (which is myjob.log above).

Next, submit this using:

qsub myjob.sge

and await an email to notify you that the job has completed.

Installation instructions

This section is primarily of interest

Installation notes

This section is primarily of interest to system administrators.



Version 5.2

No installation notes are available.

This modulefile was installed as /usr/local/modulefiles/apps/binapps/comsol/5.2.

ctffind

ctffind

Versions 4.1.5 3.140609

URL http://grigoriefflab.janelia.org/ctf

CTFFIND4, CTFFIND3 and CTFTILT are programs for finding CTFs of electron micrographs. The programsCTFFIND3 and CTFFIND4 are updated versions of the program CTFFIND2, which was developed in 1998 by Niko-laus Grigorieff at the MRC Laboratory of Molecular Biology in Cambridge, UK with financial support from the MRC.This software is licensed under the terms of the Janelia Research Campus Software Copyright 1.1.

Making ctffind available

To activate and use CTFFIND4, run:

module load apps/gcc/4.9.2/ctffind/4.1.5

This makes the following programs available in your session:

• ctffind

• ctffind_plot_results.sh

To activate and use CTFFIND3, run:

module load apps/gcc/4.4.7/ctffind/3.140609

This makes the following programs available in your session:

• ctffind3.exe

• ctffind3_mp.exe

• ctftilt.exe

• ctftilt_mp.exe

CTFFIND4 should run significantly faster than CTFFIND3 and may give slightly improved results when processingdata from detectors other than scanned photographic film.

Installation notes


Version 4.1.5

This version was built using GCC 4.9.2, the Intel Math Kernel Library (MKL) for fast fourier transforms and thewxWidgets toolkit for its user interface.

2.2. Iceberg 83

http://grigoriefflab.janelia.org/ctf

http://license.janelia.org/license/


It also has run-time dependencies on all three of those software packages.

It was installed as an optional dependency of Relion 2.

1. Download, patch, configure, compile and install using this script

2. Install this modulefile as /usr/local/modulefiles/apps/gcc/4.9.2/ctffind/4.1.5

Version 3.140609

This version was built using GCC 4.4.7.

1. Download, configure, compile and install using install_ctffind.sh

2. Install this modulefile as /usr/local/modulefiles/apps/gcc/4.4.7/ctffind/3.140609

Cufflinks

Cufflinks

Version 2.2.1

URL http://cole-trapnell-lab.github.io/cufflinks

Cufflinks assembles transcripts, estimates their abundances, and tests for differential expression and regulation inRNA-Seq samples. It accepts aligned RNA-Seq reads and assembles the alignments into a parsimonious set of tran-scripts. Cufflinks then estimates the relative abundances of these transcripts based on how many reads support eachone, taking into account biases in library preparation protocols.

Interactive Usage


The latest version of Cufflinks (currently 2.2.1) is made available with the command

module load apps/binapps/cufflinks


module load apps/binapps/cufflinks/2.2.1

This command makes the cufflinks binary directory available to your session by adding it to the PATH environmentvariable.

Documentation

The Cufflinks manual is available online at http://cole-trapnell-lab.github.io/cufflinks/manual/

Installation notes

A binary install was used


http://cole-trapnell-lab.github.io/cufflinks

http://cole-trapnell-lab.github.io/cufflinks/manual/


tar -xvzf cufflinks-2.2.1.Linux_x86_64.tar.gzmkdir -p /usr/local/packages6/apps/binapps/cufflinksmv cufflinks-2.2.1.Linux_x86_64 /usr/local/packages6/apps/binapps/cufflinks/2.2.1

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/cufflinks/2.2.1

Its contents are

## cufflinks 2.2.1 modulefile##


proc ModulesHelp { } {puts stderr "Makes cufflinks 2.2.1 available"

}

set version 2.2.1set CUFF_DIR /usr/local/packages6/apps/binapps/cufflinks/$version

module-whatis "Makes cufflinks 2.2.1 available"

prepend-path PATH $CUFF_DIR

ffmpeg

ffmpeg

Version 2.8.3

URL https://www.ffmpeg.org/

FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and playpretty much anything that humans and machines have created. It supports the most obscure ancient formats up to thecutting edge. No matter if they were designed by some standards committee, the community or a corporation. It isalso highly portable: FFmpeg compiles, runs, and passes our testing infrastructure FATE across Linux, Mac OS X,Microsoft Windows, the BSDs, Solaris, etc. under a wide variety of build environments, machine architectures, andconfigurations.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. The latest version of ffmpeg (currently 2.8.3) is made available with the command

module load apps/gcc/5.2/ffmpeg


2.2. Iceberg 85

https://www.ffmpeg.org/


module load apps/gcc/5.2/ffmpeg/2.8.3

This command makes the ffmpeg binaries available to your session. It also loads version 5.2 of the gcc compilerenvironment since gcc 5.2 was used to compile ffmpeg 2.8.3

You can now run ffmpeg. For example, to confirm the version loaded

ffmpeg -version

and to get help

ffmpeg -h

Documentation

Once you have made ffmpeg available to the system using the module command above, you can read the man pagesby typing

man ffmpeg

Installation notes

ffmpeg was installed using gcc 5.2


tar xf ./ffmpeg-2.8.3.tar.xzcd ffmpeg-2.8.3mkdir -p /usr/local/packages6/apps/gcc/5.2/ffmpeg/2.8.3./configure --prefix=/usr/local/packages6/apps/gcc/5.2/ffmpeg/2.8.3makemake install

Testing

The test suite was executed

make check

All tests passed.

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/ffmpeg/2.8.3




Flame

Flame

Version 0.17.1-openmpi-1.8.8

Dependancies gcc/4.8.2 openmpi/1.8.8

URL https://github.com/FLAME-HPC/xparser/archive/0.17.1.tar.gz

Documentation http://flame.ac.uk/docs/

The FLAME framework is an enabling tool to create agent-based models that can be run on high performance com-puters (HPCs). Models are created based upon a model of computation called (extended finite) state machines. Bydefining agent-based models in this way the FLAME framework can automatically generate simulation programs thatcan run models efficiently on HPCs. The behaviour model is based upon state machines which are composed of anumber of states with transition functions between those states. There is a single start state and by traversing statesusing the transition functions the machine executes the functions until it reaches an end state. This happens to eachagent/machine as one time step or iteration is completed

Usage

Flame can be activated using the module file:

module load apps/gcc/4.8.2/flame/0.17.1-openmpi-1.8.8

Note: The module file also loads libtool, libmboard, and the compilers gcc/4.8.3 & openmpi/1.8.8. Libmboard is thecommunication library used by Flame simulation programs.

Running Flame in parallel on multiple cores requires the batch script to be placed within the model folder. The batchscript must contain the following instructions

#!/bin/bash#$ -pe openmpi-ib no_cores#$ -cwd#$ -j y#$ -m bea#$ -M email_addressmodule load apps/gcc/4.8.2/flame/0.17.1-openmpi-1.8.8make cleancp /usr/local/packages6/apps/gcc/4.8.2/flame/0.17.1-openmpi-1.8.8/xparser/0.17.1/*.→˓tmpl .xparser model.xml -pmakempirun -np no_cores ./main no_its its/0.xml

where

no_cores = the number of computer cores required no_its = the number of time steps for the simulation 0.xml = theinitial state of the system/agents. This file is placed within the its folder within the model folder

Test

1. Download and unpack the following example models using:

2.2. Iceberg 87

https://github.com/FLAME-HPC/xparser/archive/0.17.1.tar.gz

http://flame.ac.uk/docs/


cd some_directorywget http://flame.ac.uk/docs/zip/tutorial_models.zipunzip tutorial_models.zipcd tutorial_models/model_0x

2. create the its folder within the model_0* folder, and move the 0.xml file to the its folder

3. Create a batch job submission script called test.sge containing the 12 lines of code above. Replacemodel.xml (line 10) with the actual xml model name. The script should be in the model folder.

4. Submit the job using qsub test.sge

Installation notes

Flame was compiled using the install_flame.sh script, the module file is 0.17.1-openmpi-1.8.8.

Freesurfer

Freesurfer

Versions 5.3.0

URL http://freesurfer.net/

An open source software suite for processing and analyzing (human) brain MRI images.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive graphical session with the qshcommand.

The latest version of Freesurfer is made available with the command

module load apps/binapps/freesurfer


module load apps/binapps/freesurfer/5.3.0

Important note

Freesurfer is known to produce differing results when used across platforms. See http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234 for details. We recommend that you run all of your analyses for any givenstudy on the same operating system/hardware combination and record details of these as part of your results.

Testing

No formal test suite was found. A set of commands to try were suggested at https://surfer.nmr.mgh.harvard.edu/fswiki/TestingFreeSurfer (Accessed November 6th 2015)


http://freesurfer.net/

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234

https://surfer.nmr.mgh.harvard.edu/fswiki/TestingFreeSurfer

https://surfer.nmr.mgh.harvard.edu/fswiki/TestingFreeSurfer


The following were tried

freeview -v $SUBJECTS_DIR/bert/mri/brainmask.mgz -v $SUBJECTS_DIR/bert/mri/aseg.→˓mgz:colormap=lut:opacity=0.2 -f $SUBJECTS_DIR/bert/surf/lh.white:edgecolor=yellow -→˓f $SUBJECTS_DIR/bert/surf/rh.white:edgecolor=yellow -f $SUBJECTS_DIR/bert/surf/lh.→˓pial:annot=aparc:edgecolor=red -f $SUBJECTS_DIR/bert/surf/rh.→˓pial:annot=aparc:edgecolor=red

tkmedit bert orig.mgz

tkmedit bert norm.mgz -segmentation aseg.mgz $FREESURFER_HOME/FreeSurferColorLUT.txt

tksurfer bert rh pial

qdec

Installation notes

These are primarily for administrators of the system. The github issue for the original install request is at https://github.com/rcgsheffield/sheffield_hpc/issues/164

Freesurfer was installed as follows

wget -c ftp://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/5.3.0/freesurfer-Linux-→˓centos6_x86_64-stable-pub-v5.3.0.tar.gztar -xvzf ./freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar.gzmv freesurfer 5.3.0mkdir -p /usr/local/packages6/apps/binapps/freesurfermv ./5.3.0/ /usr/local/packages6/apps/binapps/freesurfer/

The license file was obtained from the vendor and placed in /usr/local/packages6/apps/binapps/freesurfer/license.txt.The vendors were asked if it was OK to use this license on a central system. The answer is ‘yes’ - details at http://www.mail-archive.com/[email protected]/msg43872.html

To find the information necessary to create the module file

env >base.envsource /usr/local/packages6/apps/binapps/freesurfer/5.3.0/SetUpFreeSurfer.shenv > after.envdiff base.env after.env

The script SetUpFreeSurfer.sh additionally creates a MATLAB startup.m file in the user’s home directory if it doesnot already exist. This is for MATLAB support only, has not been replicated in the module file and is currently notsupported in this install.

The module file is at /usr/local/modulefiles/apps/binapps/freesurfer/5.3.0

#%Module10.2#####################################################################


# Freesurfer version (not in the user's environment)set ver 5.3.0

proc ModulesHelp { } {

2.2. Iceberg 89



http://www.mail-archive.com/[email protected]/msg43872.html

http://www.mail-archive.com/[email protected]/msg43872.html


global ver

puts stderr "Makes Freesurfer $ver available to the system."}

module-whatis "Sets the necessary Freesurfer $ver paths"

prepend-path PATH /usr/local/packages6/apps/binapps/freesurfer/$ver/binprepend-path FREESURFER_HOME /usr/local/packages6/apps/binapps/freesurfer/$ver

# The following emulates the results of 'source $FREESURFER_HOME/SetUpFreeSurfer.csh'setenv FS_OVERRIDE 0setenv PERL5LIB /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni/lib/perl5/5.8.5setenv OS Linuxsetenv LOCAL_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/localsetenv FSFAST_HOME /usr/local/packages6/apps/binapps/freesurfer/5.3.0/fsfastsetenv MNI_PERL5LIB /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni/lib/perl5/→˓5.8.5setenv FMRI_ANALYSIS_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/fsfastsetenv FSF_OUTPUT_FORMAT nii.gzsetenv MINC_BIN_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni/binsetenv SUBJECTS_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/subjects

prepend-path PATH /usr/local/packages6/apps/binapps/freesurfer/5.3.0/fsfast/bin:/usr/→˓local/packages6/apps/binapps/freesurfer/5.3.0/tktools:/usr/local/packages6/apps/→˓binapps/freesurfer/5.3.0/mni/bin

setenv FUNCTIONALS_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/sessionssetenv MINC_LIB_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni/libsetenv MNI_DIR /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni#setenv FIX_VERTEX_AREA #How do you set this to the empty string? This was done in→˓the original script.setenv MNI_DATAPATH /usr/local/packages6/apps/binapps/freesurfer/5.3.0/mni/data

gemini

gemini

Versions 0.18.0

URL http://gemini.readthedocs.org/en/latest/

GEMINI (GEnome MINIng) is a flexible framework for exploring genetic variation in the context of the wealth ofgenome annotations available for the human genome. By placing genetic variants, sample phenotypes and geno-types, as well as genome annotations into an integrated database framework, GEMINI provides a simple, flexible, andpowerful system for exploring genetic variation for disease and population genetics.

Making gemini available

The following module command makes the latest version of gemini available to your session:

module load apps/gcc/4.4.7/gemini


http://gemini.readthedocs.org/en/latest/


Alternatively, you can make a specific version available:

module load apps/gcc/4.4.7/gemini/0.18

Installation notes


• gemini was installed using the gcc 4.4.7 compiler

• install_gemini.sh

Testing

The test script used was

• test_gemini.sh

The full output from the test run is on the system at /usr/local/packages6/apps/gcc/4.4.7/gemini/0.18/test_results/

There was one failure:

effstring.t02...\cok

effstring.t03...\c15d14< gene impact impact_severity biotype is_exonic is_coding is_lof64a64> gene impact impact_severity biotype is_exonic is_coding is_loffailupdated 10 variants

annotate-tool.t1 ... okupdated 10 variants

annotate-tool.t2 ... ok

This was reported to the developers who indicated that it was nothing to worry about

Modulefile

The module file is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/gemini/0.18


#%Module1.0######################################################################### Gemini module file##



2.2. Iceberg 91

https://github.com/arq5x/gemini/issues/621


puts stderr " Adds `gemini-$geminiversion' to your PATH environment variable"}

set geminiversion 0.18

prepend-path PATH /usr/local/packages6/apps/gcc/4.4.7/gemini/0.18/bin/

GROMACS

GROMACS

Latest version 5.1

Dependancies mpi/intel/openmpi/1.10.0

URL http://www.gromacs.org/

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. To make GROMACS available in this session, run one of the following command:

source /usr/local/packages6/apps/intel/15/gromacs/5.1/bin/GMXRC

Installation notes

Version 5.1

Compilation Choices:

• Use latest intel compilers

• -DGMX_MPI=on

• -DCMAKE_PREFIX_PATH=/usr/local/packages6/apps/intel/15/gromcas

• -DGMX_FFT_LIBRARY="fftw3"

• -DGMX_BUILD_OWN_FFTW=ON

The script used to build gromacs can be found here.

htop

htop

Versions 2.0

URL http://hisham.hm/htop/


http://www.gromacs.org/

http://hisham.hm/htop/


This is htop, an interactive process viewer for Unix systems.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrsh or qshcommand.

The latest version of htop (currently 2.0) is made available with the command

module load apps/gcc/4.4.7/htop


module load apps/gcc/4.4.7/htop/2.0

This command makes the htop binary available to your session.

Installation notes

htop was installed using gcc 4.4.7

tar -xvzf ./htop-2.0.0.tar.gzcd htop-2.0.0mkdir -p /usr/local/packages6/apps/gcc/4.4.7/htop/2.0./configure --prefix=/usr/local/packages6/apps/gcc/4.4.7/htop/2.0makemake install

Testing


Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/htop/2.0


IDL

IDL

Version 8.5

Support Level extras

Dependancies java

URL http://www.exelisvis.co.uk/ProductsServices/IDL.aspx

Documentation http://www.exelisvis.com/docs/using_idl_home.html

2.2. Iceberg 93

http://www.exelisvis.co.uk/ProductsServices/IDL.aspx

http://www.exelisvis.com/docs/using_idl_home.html


IDL is a data analysis language that first appeared in 1977.

Usage

If you wish to use the IDLDE then you may need to request more memory for the interactive session using somethinglike qsh -l rmem=8G.

IDL versions can be activated using specific module files:

module load apps/idl/8.5module load apps/idl/8.5sswmodule load apps/idl/8.4

then run using idl or idlde for the interactive development environment. Note apps/idl/8.5ssw is the SolarSoft IDLenvironment.

The IDL licence is restricted - please contact us via helpdesk if you need to use this software.

Installation notes

Extract the supplied linux x86-64 tar file, run the install shell script and point it at the install directory.

Integrative Genomics Viewer (IGV)

Integrative Genomics Viewer (IGV)

Version 2.3.63

URL https://www.broadinstitute.org/igv/

The Integrative Genomics Viewer (IGV) is a high-performance visualization tool for interactive exploration of large,integrated genomic datasets. It supports a wide variety of data types, including array-based and next-generationsequence data, and genomic annotations.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an graphical interactive session with the qshcommand. In testing, we determined that you need to request at least 7 Gigabytes of real memory to launch IGV

qsh -l rmem=7G

The latest version of IGV (currently 2.3.63) is made available with the command

module load apps/binapps/igv


module load apps/binapps/igv/2.3.63

This command makes the IGV binary directory available to your session by adding it to the PATH environmentvariable. Launch the applications with the command


https://www.broadinstitute.org/igv/


igv.sh

Documentation

The IGV user guide is available online at https://www.broadinstitute.org/igv/UserGuide

Installation notes

A binary install was used

unzip IGV_2.3.63.zipmkdir -p /usr/local/packages6/apps/binapps/IGVmv ./IGV_2.3.63 /usr/local/packages6/apps/binapps/IGV/2.3.63rm /usr/local/packages6/apps/binapps/IGV/2.3.63/igv.bat

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/igv/2.3.63

Its contents

#%Module1.0######################################################################### IGV 2.3.63 modulefile##


proc ModulesHelp { } {puts stderr "Makes IGV 2.3.63 available"

}

set version 2.3.63set IGV_DIR /usr/local/packages6/apps/binapps/IGV/$version

module-whatis "Makes IGV 2.3.63 available"

prepend-path PATH $IGV_DIR

ImageJ

ImageJ

Version 1.50g

URL http://imagej.nih.gov/ij/

ImageJ is a public domain, Java-based image processing program developed at the National Institutes of Health.

2.2. Iceberg 95

https://www.broadinstitute.org/igv/UserGuide

http://imagej.nih.gov/ij/


Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx com-mand.

The latest version of ImageJ (currently 1.50g) is made available with the command

module load apps/binapps/imagej


module load apps/binapps/imagej/1.50g

This module command also changes the environment to use Java 1.8 and Java 3D 1.5.2 An environment variable calledIMAGEJ_DIR is created by the module command that contains the path to the requested version of ImageJ.

You can start ImageJ, with default settings, with the command

imagej

This starts imagej with 512Mb of Java heap space. To request more, you need to run the Java .jar file directly and usethe -Xmx Java flag. For example, to request 1 Gigabyte

java -Xmx1G -Dplugins.dir=$IMAGEJ_DIR/plugins/ -jar $IMAGEJ_DIR/ij.jar

Documentation

Links to documentation can be found at http://imagej.nih.gov/ij/download.html

Installation notes

Install version 1.49 using the install_imagej.sh script

Run the GUI and update to the latest version by clicking on Help > Update ImageJ

Rename the run script to imagej since run is too generic.

Modify the imagej script so that it reads:

java -Xmx512m -jar -Dplugins.dir=/usr/local/packages6/apps/binapps/imagej/1.50g/→˓plugins/ /usr/local/packages6/apps/binapps/imagej/1.50g/ij.jar

ImageJ’s 3D plugin requires Java 3D which needs to be included in the JRE used by imagej. I attempted a module-controlled version of Java3D but the plugin refused to recognise it. See https://github.com/rcgsheffield/sheffield_hpc/issues/279 for details.

The plug-in will install Java3D within the JRE for you if you launch it and click Plugins>3D>3D Viewer

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/imagej/1.50g

Its contents are:


http://imagej.nih.gov/ij/download.html




#%Module1.0######################################################################### ImageJ 1.50g modulefile##


module load apps/java/1.8u71

proc ModulesHelp { } {puts stderr "Makes ImageJ 1.50g available"

}

set version 1.50gset IMAGEJ_DIR /usr/local/packages6/apps/binapps/imagej/$version

module-whatis "Makes ImageJ 1.50g available"

prepend-path PATH $IMAGEJ_DIRprepend-path CLASSPATH $IMAGEJ_DIR/prepend-path IMAGEJ_DIR $IMAGEJ_DIR

ImageMagick

ImageMagick

Version 6.7.2.7-5

URL https://legacy.imagemagick.org/script/index.php

ImageMagick is a collection of tools for creating, editing, viewing, composing and converting images. Many bitmapand vector-based file formats are supported.

ImageMagick is installed on the cluster’s worker nodes. It provides a number of command-line programs plus an APIfor programmatic control.

Command-line usage

Some of the some of the most useful included command-line programs are:

display

Display an image stored in a file:

display winterview.jpg

display -resize 800x600 winterview.jpg

Display a series of files as a slide show:

2.2. Iceberg 97

https://legacy.imagemagick.org/script/index.php


display -delay 2 *.jpg

convert

Convert an image from one format to another e.g.

convert -format tiff pic1.jpg pic1.tiff

To capture from screen to a file:

convert -format jpeg X: newpicture.jpg

animate

Create an animation effect from a set of files containing snap-shots:

animate pic1.tiff pic2.tiff pic3.tiff pic4.tiff pic5.tiff

Create an animated-gif from a series of gif files (mypics*.gif):

convert -delay 40 -loop 0 mypics*.gif myanimated.gif

Note that non-gif files should first be converted:

convert -format gif mypic1.jpg mypic1.gif

identify

View image metadata:

identify parkwood.jpg

More detailed information:

identify -verbose parkwood.jpg

More information on on these and other provided command-line tools (compare, composite, conjure, import,mogrify, montage and stream) can be found in the official ImageMagick 6 documentation.

Programmatic access (API)

There are ImageMagick APIs for most common programming languages, allowing you to script/automate your imagemanipulation using C++, Ruby, Perl, Python etc.

A list of ImageMagick APIs can be found on the ImageMagick 6 site. This list may not be comprehensive.


https://legacy.imagemagick.org/script/command-line-tools.php

https://legacy.imagemagick.org/script/develop.php


Installation notes

Version 6.7.2.7-5

This was installed on the worker nodes using the operating system’s package manager (i.e. the RPM for Centos 6.x isinstalled).

R (Intel Build)

R

Dependencies BLAS

URL http://www.r-project.org/

Documentation http://www.r-project.org/

R is a statistical computing language. This version of R is built using the Intel compilers and the Intel Math KernelLibrary. This combination can result in significantly faster runtimes in certain circumstances.

Most R extensions are written and tested for the gcc suite of compilers so it is recommended that you perform testingbefore switching to this version of R.

CPU Architecture

The Intel build of R makes use of CPU instructions that are only present on the most modern of Iceberg’s nodes. Inorder to use them, you should add the following to your batch submission scripts:

#$ -l arch=intel-e5-2650v2

If you do not do this, you will receive the following error message when you try to run R:

Please verify that both the operating system and the processor support Intel(R) F16C→˓and AVX instructions.

Loading the modules

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qrshx com-mand.

There are two types of the Intel builds of R, sequential and parallel`.` ``sequential makes use of oneCPU core and can be used as a drop-in replacement for the standard version of R installed on Iceberg.

module load apps/intel/15/R/3.3.1_sequential

The parallel version makes use of multiple CPU cores for certain linear algebra routines since it is linked to theparallel version of the Intel MKL. Note that only linear algebra routines are automatically parallelised.

module load apps/intel/15/R/3.3.1_parallel

2.2. Iceberg 99

http://www.r-project.org/



When using the parallel module, you must also ensure that you set the bash environment variableOMP_NUM_THREADS to the number of cores you require and also use the openmp parallel environment. E.g. Addthe following to your submission script:

#$ -pe openmp 8export OMP_NUM_THREADS=8


Example batch jobs

Here is how to run the R script called linear_algebra_bench.r from the HPC Examples github repository:

#!/bin/bash#This script runs the linear algebra benchmark multiple times using the intel-→˓compiled version of R#that's linked with the sequential MKL#$ -l rmem=8G# Target the Ivy Bridge Processors#$ -l arch=intel-e5-2650v2

module load apps/intel/15/R/3.3.1_sequential

echo "Intel R with sequential MKL on intel-e5-2650v2"Rscript linear_algebra_bench.r output_data.rds

Here is how to run the same code using 8 cores:

#!/bin/bash#$ -l rmem=3G # Memory per core# Target the Ivy Bridge Processors#$ -l arch=intel-e5-2650v2#$ -pe openmp 8export OMP_NUM_THREADS=8


echo "Intel R with parallel MKL on intel-e5-2650v2"echo "8 cores"Rscript inear_algebra_bench.r 8core_output_data.rds

Installing additional packages

By default, the standard version of R allows you to install packages into the location ~/R/x86_64-unknown-linux-gnu-library/3.3/, where ~ refers to your home directory.

To ensure that the Intel builds do not contaminate the standard gcc builds, the Intel R module files set the environmentvariable R_LIBS_USER to point to ~/R/intel_R/3.3.1

As a user, you should not need to worry about this detail and just install packages as you usuall would from within R.e.g.

install.packages("dplyr")


https://github.com/mikecroucher/HPC_Examples


The Intel build of R will ignore any packages installed in your home directory for the standard version of R and viceversa

Installation Notes

These notes are primarily for administrators of the system.

version 3.3.1

This was a scripted install. It was compiled from source with Intel Compiler 15.0.3 and with --enable-R-shlibenabled. It was run in batch mode.

This build required several external modules including xz utils, curl, bzip2 and zlib

Sequential

1. Install using this script. Downloads, compiles, tests and installs R 3.3.1 using Intel Compilers andthe sequential MKL. The install and test logs are at /usr/local/packages6/apps/intel/15/R/sequential-3.3.1/install_logs/

2. This module file installed as apps/intel/15/R/3.3.1_sequential

Parallel

1. Install using this script. Downloads, compiles, tests and installs R 3.3.1 using Intel Compilers andthe parallel MKL. The install and test logs are at /usr/local/packages6/apps/intel/15/R/parallel-3.3.1/install_logs/

2. This module file installed as apps/intel/15/R/3.3.1_parallel

ITK-SNAP

ITK-SNAP

Version 3.6.0-rc1

Dependancies

URL http://www.itksnap.org

Documentation http://www.itksnap.org/pmwiki/pmwiki.php?n=Documentation.SNAP3

ITK-SNAP is a software application used to segment structures in 3D medical images.

Usage

The lastest version of ITK-SNAP can be activated using the module file:

module load apps/binapps/itksnap

2.2. Iceberg 101

http://www.itksnap.org

http://www.itksnap.org/pmwiki/pmwiki.php?n=Documentation.SNAP3


Alternatively, you can load a specific version of ITK-SNAP:

module load apps/binapps/itksnap/3.6

Installation notes

ITK-SNAP was compiled using the install_itksnap.sh script, the module file is 3.6.

JAGS

JAGS

Latest version 4.2

URL http://mcmc-jags.sourceforge.net/

JAGS is Just Another Gibbs Sampler. It is a program for analysis of Bayesian hierarchical models using Markov ChainMonte Carlo (MCMC) simulation not wholly unlike BUGS. JAGS was written with three aims in mind:

• To have a cross-platform engine for the BUGS language

• To be extensible, allowing users to write their own functions, distributions and samplers.

• To be a platform for experimentation with ideas in Bayesian modeling

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx or qrshcommand. To make JAGS available in this session, run one of the following module commands

module load apps/gcc/4.8.2/JAGS/4.2module load apps/gcc/4.8.2/JAGS/3.4module load apps/gcc/4.8.2/JAGS/3.1

You can now run the jags command

jagsWelcome to JAGS 4.2.0 on Thu Jun 30 09:21:17 2016JAGS is free software and comes with ABSOLUTELY NO WARRANTYLoading module: basemod: okLoading module: bugs: ok.

The rjags and runjags interfaces in R

rjags and runjags are CRAN packages that provide an R interface to jags. They are not installed in R by default.

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qrshx command.Run the following module commands


http://mcmc-jags.sourceforge.net/

https://cran.r-project.org/web/packages/rjags/index.html

https://cran.r-project.org/web/packages/runjags/index.html


module load compilers/gcc/4.8.2module load apps/gcc/4.8.2/JAGS/4.2module load apps/R/3.3.0

Launch R by typing R and pressing return. Within R, execute the following commands

install.packages('rjags')install.packages('runjags')

and follow the on-screen inctructions. Answer y to any questions about the creation of a personal library should theybe asked.

The packages will be stored in a directory called R within your home directory.

You should only need to run the install.packages commands once. When you log into the system in future,you will only need to run the module commands above to make JAGS available to the system.

You load the rjags packages the same as you would any other R package

library('rjags')library('runjags')

If you received an error message such as

Error : .onLoad failed in loadNamespace() for 'rjags', details:call: dyn.load(file, DLLpath = DLLpath, ...)error: unable to load shared object '/home/te1st/R/x86_64-unknown-linux-gnu-library/

→˓3.2/rjags/libs/rjags.so':libjags.so.3: cannot open shared object file: No such file or directory

Error: package or namespace load failed for 'rjags'

the most likely cause is that you forget to load the necessary modules before starting R.

Installation notes

Version 4.2

JAGS 4.2 was built with gcc 4.8.2

• Installed using install.sh

• This module file was installed as /usr/local/modulefiles/apps/gcc/4.8.2/JAGS/4.2

Version 3.4


module load compilers/gcc/4.8.2tar -xvzf ./JAGS-3.4.0.tar.gzcd JAGS-3.4.0mkdir -p /usr/local/packages6/apps/gcc/4.8.2/JAGS/3.4./configure --prefix=/usr/local/packages6/apps/gcc/4.8.2/JAGS/3.4makemake install

2.2. Iceberg 103


Version 3.1


module load compilers/gcc/4.8.2tar -xvzf ./JAGS-3.1.0.tar.gzcd JAGS-3.1.0mkdir -p /usr/local/packages6/apps/gcc/4.8.2/JAGS/3.1./configure --prefix=/usr/local/packages6/apps/gcc/4.8.2/JAGS/3.1makemake install

Java

Java

Latest Version 1.8.0u112

URL https://www.java.com/en/download/

Java is a programming language and computing platform first released by Sun Microsystems in 1995.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrsh or qshcommand.

The latest version of Java (currently 1.8.0u112) is made available with the command:

module load apps/java

Alternatively, you can load a specific version with one of

module load apps/java/1.8.0u112module load apps/java/1.8u71module load apps/java/1.7.0u55module load apps/java/1.7module load apps/java/1.6

Check that you have the version you expect. First, the runtime:

java -version

java version "1.8.0_112"Java(TM) SE Runtime Environment (build 1.8.0_112-b15)Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

Now, the compiler

javac -version

javac 1.8.0_112


https://www.java.com/en/download/


Virtual Memory

Note: The following is only relevant when revisiting older job submission scripts and documentation as the jobscheduler now polices jobs using real memory, not virtual memory.

By default, Java requests a lot of virtual memory on startup. This is usually a given fraction of the physical memoryon a node, which can be quite a lot on Iceberg. See What are the rmem (real memory) and (deprecated) mem (virtualmemory) options? for explanations of the difference between virtual and real/physical memory.

The default amount of virtual memory that Java uses at startup is controlled in two different ways on Iceberg:

For versions prior to 1.8.0u112 we created a wrapper script to java that uses the -Xmx1G switch to force Java torestrict its heap size to a maximum of 1 GB of memory. If this is amount is insufficient, you are free to allocate asmuch memory as you require by setting your own value for -Xmx.

For example, consider the following submission script:

#!/bin/bash#Request 9 gigabytes of real memory from the scheduler (mem)#$ -l rmem=9G

# load the Java modulemodule load apps/java/1.8u71

#Run java program allocating 5 Gigabytesjava -Xmx5G HelloWorld

Note that the above is not possible if an appliation (e.g. a startup script for another software package on the cluster)starts java itself (instead of you explicitly starting it) and the above also will not help if an application uses aninternally packaged version of Java (rather than one that can be activated using module load).

For versions 1.8.0u112 onwards the maximum heap size is instead to be restricted using an environment variable.The following is set when you run module load ...:

_JAVA_OPTIONS='-Xmx1G'

You can override this default value by running something like:

export _JAVA_OPTIONS='-Xmx6G'

before starting your application that depends on Java. _JAVA_OPTIONS can be interpretted by Java programs youstart and Java programs started by other programs, as well as by Java Virtual Machines (JVMs) that you activate usingmodule load and JVMs that are packaged within applications.

Installation notes

These are primarily for administrators of the system.

Version 1.8.0u112

1. Unpack the Java JDK tarball into /usr/local/packages6/apps/binapps/java/jdk1.8.0u112/

2. Install this modulefile as /usr/local/modulefiles/apps/binapps/java/1.8.0u112.

2.2. Iceberg 105


Note that the modulefile contains the following line:

setenv _JAVA_OPTIONS -Xmx1G

Versions prior to 1.8.0u112

Unzip and copy the install directory to /usr/local/packages6/apps/binapps/java/jdk${VERSION}/

To fix the virtual memory issue described above, we use a wrapper around the java install that sets Java’s maximumheap size (Xmx) parameter to a reasonable value.

Create the file /usr/local/packages6/apps/binapps/java/jdk1.8.0_71/shef/java with con-tents:

#!/bin/bash## Java version 1.8 cannot be invoked without specifying the java virtual# machine size due to the limitations imposed by us via SGE on memory usage.# Therefore this script intercepts the java invocations and adds a# memory constraint parameter to java engine unless there was one already# specified on the command parameter.##

if test -z "ècho $* | grep -e -Xmx`"; then# user has not specified -Xmx memory requirement flag, so add it.

/usr/local/packages6/apps/binapps/java/jdk1.8.0_71/bin/java -Xmx1G $*else# user specified the -Xmx flag, so don't add it.

/usr/local/packages6/apps/binapps/java/jdk1.8.0_71/bin/java $*fi

The module file is at /usr/local/modulefiles/apps/java/1.8u71. Its contents are

#%Module10.2#####################################################################


proc ModulesHelp { } {global helpmsgputs stderr "\t$helpmsg\n"

}

set version 1.8

set javahome /usr/local/packages6/apps/binapps/java/jdk1.8.0_71/

if [ file isdirectory $javahome/bin ] {module-whatis "Sets JAVA to version $version"set helpmsg "Changes the default version of Java to Version $version"# bring in new versionsetenv JAVA_HOME $javahomeprepend-path PATH $javahome/bin



prepend-path PATH $javahome/shefprepend-path MANPATH $javahome/man

} else {module-whatis "JAVA $version not installed"set helpmsg "JAVA $version not installed"if [ expr [ module-info mode load ] || [ module-info mode display ] ] {

# bring in new versionputs stderr "JAVA $version not installed on [uname nodename]"

}}

Julia

Julia

Versions 0.5.0-rc3

URL http://julialang.org/

Julia is a high-level, high-performance dynamic programming language for technical computing, with syntax that isfamiliar to users of other technical computing environments.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with either the or qrsh orqrshx commands.

The latest version of Julia (currently 0.5.0-rc3) is made available with the commands

module load compilers/gcc/5.2module load apps/gcc/5.2/julia/0.5.0-rc3

This adds Julia to your PATH and also loads the gcc 5.2 compiler environment with which Julia was built.

Start Julia by executing the command

julia

You can exit a Julia session with the quit() function.

Installation notes


Julia was installed using gcc 5.2

module load apps/gcc/5.2/git/2.5module load compilers/gcc/5.2module load apps/python/anaconda2-2.5.0

git clone git://github.com/JuliaLang/julia.gitcd julia

2.2. Iceberg 107

http://julialang.org/


git checkout release-0.5#The next line targets the NEHALEM CPU architecture. This is the lowest architecture→˓available on#Iceberg and so the resulting binary will be supported on all nodes. Performance will→˓not be as good as it#could be on modern nodes.sed -i s/OPENBLAS_TARGET_ARCH:=/OPENBLAS_TARGET_ARCH:=NEHALEM/ ./Make.incmake

Module File

The module file is as below

#%Module10.2#####################################################################


proc ModulesHelp { } {global ver

puts stderr " Adds Julia $ver to your environment variables."}

# Mathematica version (not in the user's environment)set ver 0.5.0-rc3

module-whatis "sets the necessary Julia $ver paths"

prepend-path PATH /usr/local/packages6/apps/gcc/5.2/julia/0.5.0-rc3

Knime

Knime

URL https://www.knime.org

Documentation http://tech.knime.org/documentation

KNIME Analytics Platform is an open solution for data-driven innovation, helping you discover the potential hiddenin your data, mine for fresh insights, or predict new futures.

Interactive Usage


The latest version of Knime including all of the extensions can be loaded with


https://www.knime.org

http://tech.knime.org/documentation


module load apps/knime

Alternatively, you can load a specific version of Knime excluding all of the extensions using one of the following

module load apps/knime/3.1.2

With extensions

module load apps/knime/3.1.2ext

Knime can then be run with

$ knime

Batch usage

There is a command line option allowing the user to run KNIME in batch mode:

knime -application org.knime.product.KNIME_BATCH_APPLICATION -nosplash -workflowDir="→˓<path>"

• -application org.knime.product.KNIME_BATCH_APPLICATION launches the KNIME batch application.

• -nosplash does not show the initial splash window.

• -workflowDir=”<path>” provides the path of the directory containing the workflow.

Full list of options:

• -nosave do not save the workflow after execution has finished

• -reset reset workflow prior to execution

• -failonloaderror don’t execute if there are errors during workflow loading

• -updateLinks update meta node links to latest version

• -credential=name[;login[;password]] for each credential enter credential name and optional login/password

• -masterkey[=...] prompt for master password (used in e.g. database nodes),if provided with argument, useargument instead of prompting

• -preferences=... path to the file containing eclipse/knime preferences,

• -workflowFile=... ZIP file with a ready-to-execute workflow in the root of the ZIP

• -workflowDir=... directory with a ready-to-execute workflow

• -destFile=... ZIP file where the executed workflow should be written to if omitted the workflow is only saved inplace

• -destDir=... directory where the executed workflow is saved to if omitted the workflow is only saved in place

• -workflow.variable=name,value,type define or overwrite workflow variable’name’ with value ‘value’ (possiblyenclosed by quotes). The’type’ must be one of “String”, “int” or “double”.

The following return codes are defined:

• 0 upon successful execution

• 2 if parameters are wrong or missing

• 3 when an error occurs during loading a workflow

2.2. Iceberg 109


• 4 if an error during execution occurred

Installation Notes


Version 3.1.2 without extensions

• Download with wget https://download.knime.org/analytics-platform/linux/knime_3.1.2.linux.gtk.x86_64.tar.gz

• Move to /usr/local/extras/knime_analytics/3.1.2

• Unzip tar -xvzf knime_3.1.2.linux.gtk.x86_64.tar.gz

The modulefile is at /usr/local/extras/modulefiles/apps/knime/3.1.2

contains

#%Module1.0

proc ModulesHelp { } {puts stderr " Adds KNIME to your PATH environment variable and necessary

→˓libraries"}

prepend-path PATH /usr/local/extras/knime_analytics/3.1.2

Version 3.1.2 with extensions

• Download with wget https://download.knime.org/analytics-platform/linux/knime-full_3.1.2.linux.gtk.x86_64.tar.gz

• Move to /usr/local/extras/knime_analytics/3.1.2ext

• Unzip tar -xvzf knime-full_3.1.2.linux.gtk.x86_64.tar.gz

The modulefile is at /usr/local/extras/modulefiles/apps/knime/3.1.2ext

contains

#%Module1.0

proc ModulesHelp { } {puts stderr " Adds KNIME to your PATH environment variable and necessary

→˓libraries"}

prepend-path PATH /usr/local/extras/knime_analytics/3.1.2ext

Lua

Lua

URL https://www.lua.org/

Documentation https://www.lua.org/docs.html


https://www.lua.org/

https://www.lua.org/docs.html


Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.

Interactive Usage


Lua can be loaded with

module load apps/lua/5.3.3

Lua can then be run with

$ lua

Installation Notes


Version 5.3.3

• curl -R -O http://www.lua.org/ftp/lua-5.3.3.tar.gz

• tar zxf lua-5.3.3.tar.gz

• cd lua-5.3.3

• make linux test

The modulefile is at /usr/local/extras/modulefiles/apps/lua/5.3.3

contains

#%Module1.0

proc ModulesHelp { } {puts stderr " Adds Lua to your PATH environment variable and necessary

→˓libraries"}

prepend-path PATH /usr/local/extras/Lua/lua-5.3.3/src

Macaulay2

Macaulay2

Latest version 1.9.2

Dependancies mpi/intel/openmpi/1.10.0 compilers/gcc/5.3 libs/gcc/5.2/boost/1.59libs/gcc/lapack/3.3.0 apps/gcc/4.4.7/xzutils/5.2.2

URL http://www.math.uiuc.edu/Macaulay2/

2.2. Iceberg 111

http://www.lua.org/ftp/lua-5.3.3.tar.gz

http://www.math.uiuc.edu/Macaulay2/


Macaulay2 is a software system devoted to supporting research in algebraic geometry and commutative algebra.

Macaulay2 includes core algorithms for computing Gröbner bases and graded or multi-graded free resolutions of mod-ules over quotient rings of graded or multi-graded polynomial rings with a monomial ordering. The core algorithmsare accessible through a versatile high level interpreted user language with a powerful debugger supporting the creationof new classes of mathematical objects and the installation of methods for computing specifically with them.

Macaulay2 can compute Betti numbers, Ext, cohomology of coherent sheaves on projective varieties, primary decom-position of ideals, integral closure of rings, and more.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. To make Macaulay2 available in this session, run one of the following command:

module load apps/gcc/5.3/macaulay2/1.9.2

You should then be able to start an interactive Macaulay2 session using

M2

Testing

Macaulay2 can perform an extensive series of self-tests at compile-time (by running make check). These tests wereperformed and passed when building Macaulay2 on Iceberg.

Installation notes and modulefile

Version 1.9.2

• Built using install.sh

• Built using a specific git commit (aedebfc1e6326416cb01598e09e8b4dbdc76c178) rather than the tagged 1.9.2release as Macaulay2’s self-tests failed for the latter. There are few differences between this commit and thetagged 1.9.2 release.

• Made available to users via this modulefile located on the system at /usr/local/modulefiles/apps/gcc/5.3/macaulay2/1.9.2

• The install script and module file enable several other module files to provide:

– OpenMPI (mpi/intel/openmpi/1.10.0)

– GCC’s compiler and Fortran library (compilers/gcc/5.3)

– the Boost C++ libraries (libs/gcc/5.2/boost/1.59)

– LAPACK linear algebra routines (libs/gcc/lapack/3.3.0)

– LZMA (xz) (de)compression (apps/gcc/4.4.7/xzutils/5.2.2)


http://en.wikipedia.org/wiki/Algebraic_geometry

http://en.wikipedia.org/wiki/Commutative_algebra

http://www.math.uiuc.edu/Macaulay2/Documentation/___Gr�bner_spbases.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_resolution_lp__Module_rp.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_modules.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_modules.html

http://www.math.uiuc.edu/Macaulay2/Documentation/

http://www.math.uiuc.edu/Macaulay2/Documentation/_graded_spand_spmultigraded_sppolynomial_springs.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_polynomial_springs.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_monomial_sporderings.html

http://www.math.uiuc.edu/Macaulay2/Documentation/___The_sp__Macaulay2_splanguage.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_the_spdebugger.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_making_spnew_spclasses.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_what_spa_spclass_spis.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_installing_spmethods.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_betti_lp__Graded__Module_rp.html

http://www.math.uiuc.edu/Macaulay2/Documentation/___Ext.html

http://www.math.uiuc.edu/Macaulay2/Documentation/___H__H%5E__Z__Z_sp__Coherent__Sheaf.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_primary_spdecomposition.html

http://www.math.uiuc.edu/Macaulay2/Documentation/_primary_spdecomposition.html

http://www.math.uiuc.edu/Macaulay2/doc/Macaulay2-1.9.2/share/doc/Macaulay2/IntegralClosure/html/

http://www.math.uiuc.edu/Macaulay2/Documentation/_rings.html

http://www.math.uiuc.edu/Macaulay2/Documentation/


Maple

Maple

Latest Version 2016

URL http://www.maplesoft.com/products/maple/

Scientific Computing and Visualisation

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qrshx command.

The latest version of Maple (currently 2016) is made available with the command

module load apps/binapps/maple


module load apps/binapps/maple/2016module load apps/binapps/maple/2015

You can then run the graphical version of Maple by entering xmaple or the command line version by entering maple.

Batch usage

It is not possible to run Maple worksheets in batch mode. Instead, you must convert your worksheet to a pure text filethat contains a set of maple input commands. You can do this in Maple by opening your worksheet and clicking onFile->Export As->Maple Input. The result will have the file extension .mpl

An example Sun Grid Engine submission script that makes use of a .mpl file called, for example, mycode.mpl is

#!/bin/bash# Request 4 gigabytes of real memory#$ -l rmem=4G

module load apps/binapps/maple/2015

maple < mycode.mpl

For general information on how to submit batch jobs refer to Running batch jobs

Tutorials

• High Performance Computing with Maple A tutorial from the Sheffield Research Software Engineering groupon how to use Maple in a High Performance Computing environment

Installation notes


2.2. Iceberg 113

http://www.maplesoft.com/products/maple/

http://rse.shef.ac.uk/blog/HPC-Maple-1/


Maple 2016

• Run the installer Maple2016.1LinuxX64Installer.run and follow instructions.

• Choose Install folder /usr/local/packages6/apps/binapps/maple2016

• Do not configure MATLAB

• Choose a network license. Details on CiCS internal wiki.

• Uncheck ‘Enable periodic checking for Maple 2016 updates’

• Check ‘Check for updates now’

The module file is at /usr/local/modulefiles/apps/binapps/maple/2016

#%Module10.2#####################################################################



puts stderr " Makes Maple $ver available to the system."}

# Maple version (not in the user's environment)set ver 2016

module-whatis "sets the necessary Maple $ver paths"

prepend-path PATH /usr/local/packages6/apps/binapps/maple2016/bin

Maple 2015

The module file is at /usr/local/modulefiles/apps/binapps/maple/2015

#%Module10.2#####################################################################



puts stderr " Makes Maple $ver available to the system."}

# Maple version (not in the user's environment)set ver 2015

module-whatis "sets the necessary Maple $ver paths"

prepend-path PATH /usr/local/packages6/maple/bin/



Mathematica

Wolfram Mathematica

Dependancies None

URL http://www.wolfram.com/mathematica/


Mathematica is a technical computing environment and programming language with strong symbolic and numericalabilities.

Single Core Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with qrshx.

The latest version of Mathematica can be loaded with

module load apps/binapps/mathematica

Alternatively, you can load a specific version of Mathematica using

module load apps/binapps/mathematica/11.0.1module load apps/binapps/mathematica/10.3.1module load apps/binapps/mathematica/10.2

Mathematica can then be started with the mathematica command

mathematica

Multicore Interactive Usage

Mathematica has extensive parallel functionality. To use it, you should request a parallel interactive session. Forexample, to request 4 cores

qrshx -pe openmp 4

Load and launch Mathematica

module load apps/binapps/mathematicamathematica

In Mathematica, let’s time how long it takes to calculate the first 20 million primes on 1 CPU core

AbsoluteTiming[primelist = Table[Prime[k], {k, 1, 20000000}];]

When I tried this, I got 78 seconds. Your results may vary greatly. Now, let’s launch 4 ParallelKernels and redo thecalculation in parallel

LaunchKernels[4]AbsoluteTiming[primelist =

ParallelTable[Prime[k], {k, 1, 20000000},Method -> "CoarsestGrained"];]

2.2. Iceberg 115

http://www.wolfram.com/mathematica/


When I tried this, I got 29 seconds – around 2.7 times faster. This illustrates a couple of points:-

• You should always request the same number of kernels as you requested in your qrshx command (in this case,4). If you request more, you will damage performance for yourself and other users of the system.

• N kernels doesn’t always translate to N times faster.

Batch Submission

Unfortunately, it is not possible to run Mathematica notebook .nb files directly in batch. Instead, you need to create asimple text file that contains all of the Mathematica commands you want to execute. Typically, such files are given theextension .m. Let’s run the following simple Mathematica script as a batch job.

(*Find out what version of Mathematica this machine is running*)Print["Mathematica version is " <> $Version]

(*Find out the name of the machine we are running on*)Print["The machine name is " <> $MachineName]

(*Do a calculation*)Print["The result of the integral is "]Print [ Integrate[Sin[x]^2, x]]

Copy and paste the above into a text file called very_simple_mathematica.m

An example batch submission script for this file is

#!/bin/bash# Request 4 gigabytes of real memory#$ -l rmem=4G

module load apps/binapps/mathematica/10.3.1

math -script very_simple_mathematica.m

Copy and paste the above into a file called run_job.sh and submit with

qsub run_job.sh

Once the job has successfully completed, the output will be in a file named like run_job.sh.o396699. The numberat the end refers to the job-ID given to this job by the system and will be different for you. Let’s take a look at thecontents of this file

more run_job.sh.o396699

Mathematica version is 10.2.0 for Linux x86 (64-bit) (July 28, 2015)The machine name is node131The result of the integral isx/2 - Sin[2*x]/4

Installation notes

These are primarily for administrators of the system

For Version 11.0.1



mkdir -p /usr/local/packages6/apps/binapps/mathematica/11.0.1chmod +x ./Mathematica_11.0.1_LINUX.sh./Mathematica_11.0.1_LINUX.sh

The installer is interactive. Here’s the session output

--------------------------------------------------------------------------------------→˓------------------------------------------------------------------------------------→˓----------------------------------

→˓Wolfram Mathematica 11 Installer--------------------------------------------------------------------------------------→˓------------------------------------------------------------------------------------→˓----------------------------------

Copyright (c) 1988-2016 Wolfram Research, Inc. All rights reserved.

WARNING: Wolfram Mathematica is protected by copyright law and international treaties.→˓ Unauthorized reproduction or distribution may result in severe civil and criminal→˓penalties and will beprosecuted to the maximum extent possible under law.

Enter the installation directory, or press ENTER to select /usr/local/Wolfram/→˓Mathematica/11.0:> /usr/local/packages6/apps/binapps/mathematica/11.0.1

Now installing...

[*********************************************************************************************************************************************************************************************************]

Type the directory path in which the Wolfram Mathematica script(s) will be created,→˓or press ENTER to select /usr/local/bin:> /usr/local/packages6/apps/binapps/mathematica/11.0.1/scripts

Create directory (y/n)?> y

WARNING: No Avahi Daemon was detected so some Kernel Discovery features will not be→˓available. You can install Avahi Daemon using your distribution's package→˓management system.

For Red Hat based distributions, try running (as root):

yum install avahi

Installation complete.

Copy the license from the previous version. It is a network license so you need to ensure that the network licenseserver has been updated.

cp /usr/local/packages6/apps/binapps/mathematica/10.2/Configuration/Licensing/→˓mathpass /usr/local/packages6/apps/binapps/mathematica/11.0.1/Configuration/→˓Licensing/

For Version 10.3.1

2.2. Iceberg 117


mkdir -p /usr/local/packages6/apps/binapps/mathematica/10.3.1chmod +x ./Mathematica_10.3.1_LINUX.sh./Mathematica_10.3.1_LINUX.sh


--------------------------------------------------------------------------------Wolfram Mathematica 10.3 Installer

--------------------------------------------------------------------------------


WARNING: Wolfram Mathematica is protected by copyright law and internationaltreaties. Unauthorized reproduction or distribution may result in severecivil and criminal penalties and will be prosecuted to the maximum extentpossible under law.

Enter the installation directory, or press ENTER to select/usr/local/Wolfram/Mathematica/10.3:> /usr/local/packages6/apps/binapps/mathematica/10.3.1

Now installing...

[*****************************************************************************]

Type the directory path in which the Wolfram Mathematica script(s) will becreated, or press ENTER to select /usr/local/bin:> /usr/local/packages6/apps/binapps/mathematica/10.3.1/scripts


WARNING: No Avahi Daemon was detected so some Kernel Discovery features willnot be available. You can install Avahi Daemon using your distribution'spackage management system.


yum install avahi


Install the University network mathpass file at /usr/local/packages6/apps/binapps/mathematica/10.3.1/Configuration/Licensing

For Version 10.2

mkdir -p /usr/local/packages6/apps/binapps/mathematica/10.2chmod +x ./Mathematica_10.2.0_LINUX.sh./Mathematica_10.2.0_LINUX.sh


--------------------------------------------------------------------------------------→˓------------------------------------------------------------------------------------→˓----------------------------------

→˓Wolfram Mathematica 10.2 Installer



--------------------------------------------------------------------------------------→˓------------------------------------------------------------------------------------→˓----------------------------------


WARNING: Wolfram Mathematica is protected by copyright law and international treaties.→˓ Unauthorized reproduction or distribution may result in severe civil and criminal→˓penalties and will beprosecuted to the maximum extent possible under law.

Enter the installation directory, or press ENTER to select /usr/local/Wolfram/→˓Mathematica/10.2:>

Error: Cannot create directory /usr/local/Wolfram/Mathematica/10.2.

You may need to be logged in as root to continue with this installation.

Enter the installation directory, or press ENTER to select /usr/local/Wolfram/→˓Mathematica/10.2:> /usr/local/packages6/apps/binapps/mathematica/10.2

Now installing...

[*********************************************************************************************************************************************************************************************************]

Type the directory path in which the Wolfram Mathematica script(s) will be created,→˓or press ENTER to select /usr/local/bin:> /usr/local/packages6/apps/binapps/mathematica/10.2/scripts


WARNING: No Avahi Daemon was detected so some Kernel Discovery features will not be→˓available. You can install Avahi Daemon using your distribution's package→˓management system.


yum install avahi


Remove the playerpass file

rm /usr/local/packages6/apps/binapps/mathematica/10.2/Configuration/Licensing/→˓playerpass

Install the University network mathpass file at /usr/local/packages6/apps/binapps/mathematica/10.2/Configuration/Licensing

Modulefiles

• The 11.0.1 module file.

2.2. Iceberg 119


• The 10.3.1 module file.

• The 10.2 module file.

MATLAB

MATLAB

Versions 2013a, 2013b, 2014a, 2015a, 2016a, 2016b, 2017b

Support Level FULL

Dependancies None

URL http://uk.mathworks.com/products/matlab

Local URL http://www.shef.ac.uk/wrgrid/software/matlab

Documentation http://uk.mathworks.com/help/matlab

Scientific computing and visualisation.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh command.

The latest version of MATLAB (currently 2017b) is made available with the command:

module load apps/matlab

Alternatively, you can load a specific version with one of the following commands:

module load apps/matlab/2013amodule load apps/matlab/2013bmodule load apps/matlab/2014amodule load apps/matlab/2015amodule load apps/matlab/2016amodule load apps/matlab/2016bmodule load apps/matlab/2017b

You can then run MATLAB by entering matlab

Serial (one CPU) Batch usage

Here, we assume that you wish to run the program helloworld.m on the system:

function helloworlddisp('Hello World')

end

First, you need to write a batch submission file. We assume you’ll call this my_job.sge:

#!/bin/bash#$ -l rmem=4G # Request 4 GB of real memory#$ -cwd # Run job from current directory


http://uk.mathworks.com/products/matlab

http://www.shef.ac.uk/wrgrid/software/matlab

http://uk.mathworks.com/help/matlab


module load apps/matlab/2017b # Make specific version of MATLAB availablematlab -nodesktop -r helloworld

Ensuring that helloworld.m and my_job.sge are both in your current working directory, submit your job to thebatch system:

qsub my_job.sge

Note that we are running the script helloworld.m but we drop the .m in the call to MATLAB. That is, we do -rhelloworld rather than -r helloworld.m. The output will be written to the job .o file when the job finishes.

MATLAB Compiler and running free-standing compiled MATLAB programs

The MATLAB compiler mcc can be used to generate standalone executables. These executables can then be run onother computers that do not have MATLAB installed. We strongly recommend you use R2016b or later versions totake advantage of this feature.

To compile a MATLAB function or script for example called myscript.m the following steps are required:

# Load the matlab 2017b modulemodule load apps/matlab/2017b

# Compile your program to generate the executable myscript and# also generate a shell script named run_myscript.shmcc -m myscript.m

# Finally run your program./run_myscript.sh $MCRROOT

If myscript.m is a MATLAB function that require inputs these can be suplied on the command line. For exampleif the first line of myscript.m reads:

function out = myscript ( a , b , c )

then to run it with 1.0, 2.0, 3.0 as its parameters you will need to type:

./run_myscript.sh $MCRROOT 1.0 2.0 3.0

After a successful compilation and running you can transfer your executable and the runscript to another computer.That computer does not have to have MATLAB installed or licensed on it but it will have to have the MATLAB runtimesystem installed. This can be done by either downloading the MATLAB runtime environment from Mathworks website or by copying the installer file from the cluster itself which resides in:

$MCRROOT/toolbox/compiler/deploy/glnxa64/MCRInstaller.zip

This file can be unzipped in a temporary area and run the setup script that unzipping yields to install the MATLABruntime environment. Finally the environment variable $MCRROOT can be set to the directory containing the runtimeenvironment.

Parallel MATLAB on iceberg

Currently we recommend the 2015a version of MATLAB for parallel work, and task arrays requiring more than a fewhours runtime.

2.2. Iceberg 121


The default cluster configuration named local provides parallel working environment by using the CPUs of the workernode that is running the current MATLAB session. Each iceberg worker node can run multiple users’ jobs simultane-ously. Therefore depending on who else is using that node at the time, parallel MATLAB jobs can create contentionsbetween jobs and slow them considerably. It is therefore advisable to start parallel MATLAB jobs that will use thelocal profile from a parallel SGE job. For example, to use the local profile with 5 workers, do the following;

Start a parallel OpenMP job with 6 workers:

qsh -pe smp 6

Run MATLAB in that session and select 5 workers:

matlabparpool ('local' , 5 )

The above example will use 5 MATLAB workers on a single iceberg node to run a parallel task. Note that beinggranted a multi-core interactive session by the scheduler is dependent on Iceberg’s loading at the time of request. Itis not guaranteed. However the user can reduce the number of cores requested, which will improve their chances ofbeing granted a multi-core session.

To take advantage of the multiple iceberg nodes, you will need to make use of a parallel cluster profile named sge.This can be done by issuing a locally provided MATLAB command named iceberg that imports the parallel clusterprofile named sge that can take advantage of the SGE scheduler to run larger parallel jobs.

When using the sge profile, MATLAB will be able to submit multiple MATLAB jobs the the SGE scheduler fromwithin MATLAB itself. However, each job will have the default resource requirements unless the following trick isdeployed. For example, during your MATLAB session type:

global sge_paramssge_params='-l rmem=8G -l h_rt=36:00:00'

to make sure that all the MATLAB batch jobs will use up to 8 GBytes of memory and will not be killed unless theyexceed 36 hours of run time.

Training

• CiCS run an Introduction to Matlab course

• In November 2015, CiCS hosted a masterclass in Parallel Computing in MATLAB. The materials are availableonline

Installation notes


Installation and configuration is a four-stage process:

• Set up the floating license server (the license server for earlier MATLAB versions can be used), ensuring that itcan serve licenses for any new versions of MATLAB that you want to install

• Run a graphical installer to download MATLAB archive files used by the main (automated) installation process

• Run the same installer in ‘silent’ command-line mode to perform the installation using those archive files and atext config file.

• Install a relevant modulefile

In more detail:


http://rcg.group.shef.ac.uk/courses/matlab/

http://rcg.group.shef.ac.uk/courses/mathworks-parallelmatlab/



1. If necessary, update the floating license keys on licserv4.shef.ac.uk to ensure that the licenses areserved for the versions to install.

2. Log on to Mathworks site to download the MATLAB installer package for 64-bit Linux ( for R2017b this wascalled matlab_R2017b_glnxa64.zip )

3. unzip the installer package in a directory with ~10GB of space (needed as many MATLAB archive fileswill subsequently be downloaded here). Using a directory on an NFS mount (e.g. /data/${USER}/MathWorks/R2017b) allows the same downloaded archives to be used to install MATLAB on multipleclusters.

4. ./install to start the graphical installer (needed to download the MATLAB archive files).

5. Select install choice of Log in to Mathworks Account and log in with a License Administrator account (not aLicensed End User (personal) account).

6. Select Download only.

7. Select the offered default Download path and select the directory you ran ./install from. Wait a while forall requested archive files to be downloaded.

8. Next, ensure installer_input.txt looks like the following

fileInstallationKey=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-→˓XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXXagreeToLicense=yesoutputFile=matlab_2017b_install.logmode=silentlicensePath=/usr/local/packages6/matlab/network.liclmgrFiles=falselmgrService=false

9. Create the installation directory e.g.:

mkdir -m 2755 -p /usr/local/packages6/matlab/R2017bchown ${USER}:app-admins /usr/local/packages6/matlab/R2017b

10. Run the installer using our customized installer_input.txt like so: ./install -mode silent-inputFile ${PWD}/installer_input.txt ; installation should finish with exit status 0 if all hasworked.

11. Install a modulefile with a name and path like /usr/local/modulefiles/apps/matlab/2017b andcontents like

#%Module1.0#####################################################################

## Module file loggingsource /usr/local/etc/module_logging.tcl

proc ModulesHelp { } {global versionputs stderr " Makes MATLAB 2017b available for use"

}module-whatis "Makes MATLAB 2017b available"

# Do not use other versions at the same time.conflict apps/matlab

set version 2017bset matlabroot /usr/local/packages6/matlab/R$versionprepend-path PATH $matlabroot/bin

2.2. Iceberg 123


12. Ensure the contents of the install directory and the modulefile are writable by those in app-admins groupe.g.:

chmod -R g+w ${USER}:app-admins /usr/local/packages6/matlab/R2017b /usr/local/→˓modulefiles/apps/matlab/2017b

TODO: Documentation for MATLAB parallel configuration.

MrBayes

MrBayes

Version 3.2.6

URL https://sourceforge.net/projects/mrbayes/

MrBayes is a program for the Bayesian estimation of phylogeny.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx com-mand.

The latest version of MrBayes (currently 3.2.6) is made available with the command

module load apps/gcc/4.4.7/mrbayes


module load apps/gcc/4.4.7/mrbayes/3.2.6

This command makes the MrBayes mb binary available to your session.

Installation notes

MrBayes was installed with gcc 4.4.7

tar -xvzf ./mrbayes-3.2.6.tar.gzlscd mrbayes-3.2.6autoconf./configure --with-beagle=/usr/local/packages6/libs/gcc/4.4.7/beagle/2.1.2 --prefix=/→˓usr/local/packages6/apps/gcc/4.4.7/mrbayes/3.2.6/makemake install

This modulefile was installed as /usr/local/modulefiles/apps/gcc/4.4.7/mrbayes/3.2.6


https://sourceforge.net/projects/mrbayes/


Netlogo

Netlogo

Version 5.3.1

URL https://ccl.northwestern.edu/netlogo/index.shtml

NetLogo is a multi-agent programmable modeling environment. It is used by tens of thousands of students, teachersand researchers worldwide.

Interactive Usage


The latest version of NetLogo, currently 5.3.1, is made available with the command

module load apps/binapps/netlogo/


module load apps/binapps/netlogo/5.3.1

This command makes the NetLogo executables available to your session by adding the install directory to your PATHvariable. Start the NetLogo Graphical User Interface with the command

NetLogo

Batch usage

Here is an example batch submission file for a Netlogo model that contains a BehaviourSpace experiment calledParallelExperiment1. Using BehaviourSpace is the easiest way to exploit parallel processors in Netlogo.

#!/bin/bash# Request 3 Gigabytes of RAM per core#$ -l rmem=3G#Combine stderr and stdout into one file#$ -j y#Make sure that the value below matches the number of threads#$ -pe openmp 4#Make sure this matches the number of openmp slots requestedthreads=4

module load apps/binapps/netlogo/5.3.1

#Set the filename of the output csv file#Will be called out_4.csv in this caseoutput_table=out_$threads.csv#Experiment and model namesexperiment=ParallelExperiment1model=ParallelPredation.nlogo

echo "$threads threads requested"

2.2. Iceberg 125

https://ccl.northwestern.edu/netlogo/index.shtml

https://ccl.northwestern.edu/netlogo/docs/behaviorspace.html


#You have to run netlogo from its install directory in order for extensions to workcd $NETLOGO_DIRjava -Xmx1024m -Dfile.encoding=UTF-8 -cp $NETLOGO_DIR/NetLogo.jar org.nlogo.headless.→˓Main --model $SGE_O_WORKDIR/$model --experiment $experiment --table $SGE_O_WORKDIR/→˓$output_table --threads $threads

Call the above file submit_netlogo.sh and submit it with qsub submit_netlogo.sh. All files required to run this examplecan be found in the HPC Examples repository.

Netlogo uses shared memory parallelism. As such, the maximum number of OpenMP slots can request on our currentsystem is 16.

Troubleshooting

Problems with X Windows

When you run the NetLogo command, you get the following error

NetLogo Error invoking method.NetLogo Failed to launch JVM

This could be because you have not requested an X session properly. Ensure that you have logged into the systemusing the -X flag e.g.

ssh -X [email protected]

Also ensure that you have requested a qrshx session rather than a qrsh session.

Installation notes

Download and untar the installer to /usr/local/packages6/apps/binapps/netlogo/netlogo-5.3.1-64

post installation actions

One netlogo user required the Dynamic scheduler extension. Installed by doing

wget https://github.com/colinsheppard/Dynamic-Scheduler-Extension/archive/v0.2.1.tar.→˓gztar -xvzf ./v0.2.1.tar.gzmv Dynamic-Scheduler-Extension ./dynamic-schedulermv ./dynamic-scheduler/ /usr/local/packages6/apps/binapps/netlogo/netlogo-5.3.1-64/→˓app/extensions/

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/binapps/netlogo/5.3.1

Octave


https://github.com/mikecroucher/HPC_Examples


Ocatve

Versions 4.0.0

URL https://www.gnu.org/software/octave/

GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabil-ities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. Italso provides extensive graphics capabilities for data visualization and manipulation. Octave is normally used throughits interactive command line interface, but it can also be used to write non-interactive programs. The Octave languageis quite similar to MATLAB so that most programs are easily portable.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with either the qsh or qrshcommands.

The latest version of Octave (currently 4.0.0) is made available with the command

module load apps/gcc/5.2/octave


module load apps/gcc/5.2/octave/4.0

This adds Octave to your PATH and also loads all of the supporting libraries and compilers required by the system.

Start Octave by executing the command

octave

If you are using a qsh session, the graphical user interface version will begin. If you are using a qrsh session, you willonly be able to use the text-only terminal version.

Batch Usage

Here is an example batch submission script that will run an Octave program called foo.m

#!/bin/bash# Request 5 gigabytes of real memory#$ -l rmem=5G# Request 64 hours of run time#$ -l h_rt=64:00:00

module load apps/gcc/5.2/octave/4.0

octave foo.m

Using Packages (Toolboxes)

Octave toolboxes are referred to as packages. To see which ones are installed, use the command ver from withinOctave.

2.2. Iceberg 127

https://www.gnu.org/software/octave/


Unlike MATLAB, Octave does not load all of its packages at startup. It is necessary to load the package before itscommands are available to your session. For example, as with MATLAB, the pdist command is part of the statisticspackage. Unlike MATLAB, pdist is not immediately available to you

octave:1> pdist([1 2 3; 2 3 4; 1 1 1])warning: the 'pdist' function belongs to the statistics package from OctaveForge which you have installed but not loaded. To load the package, run`pkg load statistics' from the Octave prompt.

Please read `http://www.octave.org/missing.html' to learn how you cancontribute missing functionality.warning: called from

__unimplemented__ at line 524 column 5error: 'pdist' undefined near line 1 column 1

As the error message suggests, you need to load the statistics package

octave:1> pkg load statisticsoctave:2> pdist([1 2 3; 2 3 4; 1 1 1])ans =

1.7321 2.2361 3.7417

Installation notes


Octave was installed using gcc 5.2 and the following libraries:

• Java 1.8u60

• FLTK 1.3.3

• fftw 3.3.4

• Octave was installed using a SGE batch job. The install script is on github

• The make log is on the system at /usr/local/packages6/apps/gcc/5.2/octave/4.0/make_octave4.0.0.log

• The configure log is on the system at /usr/local/packages6/apps/gcc/5.2/octave/4.0/configure_octave4.0.0.log

For full functionality, Octave requires a large number of additional libraries to be installed. We have currently notinstalled all of these but will do so should they be required.

For information, here is the relevant part of the Configure log that describes how Octave was configured

Source directory: .Installation prefix: /usr/local/packages6/apps/gcc/5.2/octave/4.0C compiler: gcc -pthread -fopenmp -Wall -W -Wshadow -Wforma

t -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wcast-align -Wcast-qual -I/usr/local/packages6/compilers/gcc/5.2.0/include

C++ compiler: g++ -pthread -fopenmp -Wall -W -Wshadow -Wold-style-cast -Wformat -Wpointer-arith -Wwrite-strings -Wcast-align -Wcast-qual -g -O2

Fortran compiler: gfortran -OFortran libraries: -L/usr/local/packages6/compilers/gcc/5.2.0/lib -

L/usr/local/packages6/compilers/gcc/5.2.0/lib64 -L/usr/local/packages6/compilers/gcc/5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0 -L/usr/local/packages6/compilers/gcc/5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/../../../../lib64 -L/lib/../



lib64 -L/usr/lib/../lib64 -L/usr/local/packages6/libs/gcc/5.2/fftw/3.3.4/lib -L/usr/local/packages6/libs/gcc/5.2/fltk/1.3.3/lib -L/usr/local/packages6/compilers/gcc/5.2.0/lib/gcc/x86_64-unknown-linux-gnu/5.2.0/../../.. -lgfortran -lm -lquadmath

Lex libraries:LIBS: -lutil -lm

AMD CPPFLAGS:AMD LDFLAGS:AMD libraries:ARPACK CPPFLAGS:ARPACK LDFLAGS:ARPACK libraries:BLAS libraries: -lblasCAMD CPPFLAGS:CAMD LDFLAGS:CAMD libraries:CARBON libraries:CCOLAMD CPPFLAGS:CCOLAMD LDFLAGS:CCOLAMD libraries:CHOLMOD CPPFLAGS:CHOLMOD LDFLAGS:CHOLMOD libraries:COLAMD CPPFLAGS:COLAMD LDFLAGS:COLAMD libraries:CURL CPPFLAGS:CURL LDFLAGS:CURL libraries: -lcurlCXSPARSE CPPFLAGS:CXSPARSE LDFLAGS:CXSPARSE libraries:DL libraries:FFTW3 CPPFLAGS:FFTW3 LDFLAGS:FFTW3 libraries: -lfftw3_threads -lfftw3FFTW3F CPPFLAGS:FFTW3F LDFLAGS:FFTW3F libraries: -lfftw3f_threads -lfftw3fFLTK CPPFLAGS: -I/usr/local/packages6/libs/gcc/5.2/fltk/1.3.3/in

clude -I/usr/include/freetype2 -I/usr/local/packages6/compilers/gcc/5.2.0/include -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_THREAD_SAFE -D_REENTRANT

FLTK LDFLAGS: -L/usr/local/packages6/libs/gcc/5.2/fltk/1.3.3/lib -Wl,-rpath,/usr/local/packages6/libs/gcc/5.2/fltk/1.3.3/lib -L/usr/local/packages6/compilers/gcc/5.2.0/lib -L/usr/local/packages6/compilers/gcc/5.2.0/lib64 -lfltk_gl -lGLU -lGL -lfltk -lXcursor -lXfixes -lXext -lXft -lfontconfig -lXinerama -lpthread -ldl -lm -lX11

FLTK libraries:fontconfig CPPFLAGS:fontconfig libraries: -lfontconfigFreeType2 CPPFLAGS: -I/usr/include/freetype2FreeType2 libraries: -lfreetypeGLPK CPPFLAGS:GLPK LDFLAGS:GLPK libraries:HDF5 CPPFLAGS:HDF5 LDFLAGS:

2.2. Iceberg 129


HDF5 libraries: -lhdf5Java home: /usr/local/packages6/apps/binapps/java/jre1.8.0_6

0/Java JVM path: /usr/local/packages6/apps/binapps/java/jre1.8.0_6

0/lib/amd64/serverJava CPPFLAGS: -I/usr/local/packages6/apps/binapps/java/jre1.8.0

_60//include -I/usr/local/packages6/apps/binapps/java/jre1.8.0_60//include/linuxJava libraries:LAPACK libraries: -llapackLLVM CPPFLAGS:LLVM LDFLAGS:LLVM libraries:Magick++ CPPFLAGS:Magick++ LDFLAGS:Magick++ libraries:OPENGL libraries: -lfontconfig -lGL -lGLUOSMesa CPPFLAGS:OSMesa LDFLAGS:OSMesa libraries:PCRE CPPFLAGS:PCRE libraries: -lpcrePortAudio CPPFLAGS:PortAudio LDFLAGS:PortAudio libraries:PTHREAD flags: -pthreadPTHREAD libraries:QHULL CPPFLAGS:QHULL LDFLAGS:QHULL libraries:QRUPDATE CPPFLAGS:QRUPDATE LDFLAGS:QRUPDATE libraries:Qt CPPFLAGS: -I/usr/include/QtCore -I/usr/include/QtGui -I/usr

/include/QtNetwork -I/usr/include/QtOpenGLQt LDFLAGS:Qt libraries: -lQtNetwork -lQtOpenGL -lQtGui -lQtCoreREADLINE libraries: -lreadlineSndfile CPPFLAGS:Sndfile LDFLAGS:Sndfile libraries:TERM libraries: -lncursesUMFPACK CPPFLAGS:UMFPACK LDFLAGS:UMFPACK libraries:X11 include flags:X11 libraries: -lX11Z CPPFLAGS:Z LDFLAGS:Z libraries: -lz

Default pager: lessgnuplot: gnuplot

Build Octave GUI: yesJIT compiler for loops: noBuild Java interface: noDo internal array bounds checking: noBuild static libraries: no



Build shared libraries: yesDynamic Linking: yes (dlopen)Include support for GNU readline: yes64-bit array dims and indexing: noOpenMP SMP multithreading: yesBuild cross tools: no

configure: WARNING:

I didn't find gperf, but it's only a problem if you need toreconstruct oct-gperf.h

configure: WARNING:

I didn't find icotool, but it's only a problem if you need toreconstruct octave-logo.ico, which is the case if you're building fromVCS sources.

configure: WARNING: Qhull library not found. This will result in loss of functionality of some geometry functions.configure: WARNING: GLPK library not found. The glpk function for solving linear programs will be disabled.configure: WARNING: gl2ps library not found. OpenGL printing is disabled.configure: WARNING: OSMesa library not found. Offscreen rendering with OpenGL will be disabled.configure: WARNING: qrupdate not found. The QR & Cholesky updating functions will be slow.configure: WARNING: AMD library not found. This will result in some lack of functionality for sparse matrices.configure: WARNING: CAMD library not found. This will result in some lack of functionality for sparse matrices.configure: WARNING: COLAMD library not found. This will result in some lack offunctionality for sparse matrices.configure: WARNING: CCOLAMD library not found. This will result in some lack offunctionality for sparse matrices.

configure: WARNING: CHOLMOD library not found. This will result in some lack offunctionality for sparse matrices.

configure: WARNING: CXSparse library not found. This will result in some lack of functionality for sparse matrices.configure: WARNING: UMFPACK not found. This will result in some lack of functionality for sparse matrices.configure: WARNING: ARPACK not found. The eigs function will be disabled.configure: WARNING: Include file <jni.h> not found. Octave will not be able tocall Java methods.configure: WARNING: Qscintilla library not found -- disabling built-in GUI editorconfigure:

• Some commonly-used packages were additionally installed from Octave Forge using the following commandsfrom within Octave

pkg install -global -forge iopkg install -global -forge statisticspkg install -global -forge mappingpkg install -global -forge imagepkg install -global -forge structpkg install -global -forge optim

2.2. Iceberg 131

http://octave.sourceforge.net/


Module File

The module file is octave_4.0

OpenFOAM

OpenFOAM

Versions 3.0.0, 2.4.0

Dependencies gcc/5.2 mpi/gcc/openmpi/1.10.0 libs/gcc/5.2/scotch/6.0.4

URL http://www.openfoam.org

Documentation http://www.openfoam.org/docs/

OpenFOAM is free, open source software for computational fluid dynamics (CFD).

Usage

You can load a specific version of OpenFOAM using one of:

module load apps/gcc/5.2/openfoam/3.0.0module load apps/gcc/5.2/openfoam/2.4.0

This has the same effect as sourcing the OpenFOAM bashrc file, so that should not be needed.

These module load commands also load:

• OpenMPI 1.10.0

• GCC 5.2

• the Scotch 6.0.4 library

Installation notes

Version 3.0.0

Compiled using this install_openfoam.sh script.

The 3.0.0 module file.

Version 2.4.0

The compilation process was not documented.

The 2.4.0 module file.


http://www.openfoam.org

http://www.openfoam.org/docs/


OpenSim

OpenSim

Support Level bronze

Dependancies None

URL https://simtk.org/home/opensim

Version 3.3

OpenSim is a freely available, user extensible software system that lets users develop models of musculoskeletalstructures and create dynamic simulations of movement.

Usage

The latest version of OpenSim can be loaded with

module load apps/gcc/4.8.2/opensim

Installation Notes


Built using:

cmake /home/te1st/Downloads/OpenSim33-source/-DCMAKE_INSTALL_PREFIX=/usr/local/packages6/apps/gcc/4.8.2/opensim/3.3/-DSIMBODY_HOME=/usr/local/packages6/libs/gcc/4.8.2/simbody/3.5.3/-DOPENSIM_STANDARD_11=ON

make -j 8

make install

orca

orca

Versions 3.0.3

URL https://orcaforum.cec.mpg.de/

ORCA is a flexible, efficient and easy-to-use general purpose tool for quantum chemistry with specific emphasis onspectroscopic properties of open-shell molecules. It features a wide variety of standard quantum chemical methodsranging from semiempirical methods to DFT to single- and multireference correlated ab initio methods. It can alsotreat environmental and relativistic effects.

2.2. Iceberg 133

https://simtk.org/home/opensim

https://orcaforum.cec.mpg.de/


Making orca available

The following module command makes the latest version of orca available to your session

module load apps/binapps/orca

Alternatively, you can make a specific version available

module load apps/binapps/orca/3.0.3

Example single core job

Create a file called orca_serial.inp that contains the following orca commands

## My first ORCA calculation :-)## Taken from the Orca manual# https://orcaforum.cec.mpg.de/OrcaManual.pdf! HF SVP

* xyz 0 1C 0 0 0O 0 0 1.13

*

Create a Sun Grid Engine submission file called submit_serial.sh that looks like this

#!/bin/bash# Request 4 Gig of real memory per process#$ -l rmem=4G

module load apps/binapps/orca/3.0.3$ORCAHOME/orca example1.inp

Submit the job to the queue with the command

qsub submit_serial.sh

Example parallel job

An example Sun Grid Engine submission script is

#!/bin/bash#Request 4 Processes#Ensure that this matches the number requested in your Orca input file#$ -pe openmpi-ib 4# Request 4 Gig of real memory per process#$ -l rmem=4G

module load mpi/gcc/openmpi/1.8.8module load apps/binapps/orca/3.0.3

ORCAPATH=/usr/local/packages6/apps/binapps/orca/3.0.3/$ORCAPATH/orca example2_parallel.inp



Register as a user

You are encouraged to register as a user of Orca at https://orcaforum.cec.mpg.de/ in order to take advantage of updates,announcements and also of the users forum.

Documentation

A comprehensive .pdf manual is available online.

Installation notes

These are primarily for system administrators. Orca was a binary install

tar -xjf ./orca_3_0_3_linux_x86-64.tbzcd orca_3_0_3_linux_x86-64mkdir -p /usr/local/packages6/apps/binapps/orca/3.0.3mv * /usr/local/packages6/apps/binapps/orca/3.0.3/

Modulefile

The module file is on the system at /usr/local/modulefiles/apps/binapps/orca/3.0.3

The contents of the module file are

#%Module1.0######################################################################### Orca module file##



puts stderr " Adds òrca-$orcaversion' to your PATH environment variable"}

set orcaversion 3.0.3prepend-path ORCAHOME /usr/local/packages6/apps/binapps/orca/3.0.3/prepend-path PATH /usr/local/packages6/apps/binapps/orca/3.0.3/

Paraview

Paraview

Version 4.3

Support Level extras

2.2. Iceberg 135


https://orcaforum.cec.mpg.de/OrcaManual.pdf


Dependancies openmpi (1.8)

URL http://paraview.org/

Documentation http://www.paraview.org/documentation/

Paraview is a parallel visualisation tool.

Using Paraview on iceberg

This guide describes how to use Paraview from iceberg. Paraview is a parallel visualisation tool designed for use onclusters like iceberg. Because iceberg is designed primarily as a headless computational cluster Paraview has not beeninstalled on iceberg in such a way that you load the GUI remotely on iceberg1. Paraview therefore runs on iceberg inclient + server mode, the server running on iceberg, and the client on your local machine. You need to:

1. Configure and start the client on your machine first then

2. then start the server on Iceberg, which will then connect back to the client.

Configure the Client

1. Download and install Paraview 4.32 on your own machine.

2. Start the paraview program on your own machine.

3. Click File -> Connect -> Add Server

4. Name: my_iceberg_paraview_config

5. Server Type: Client / Server (reverse connection) (we want Iceberg to connect to our local machine)

6. Port: 11111 (the default)

7. Click Configure to go to the next screen

8. Startup Type: Manual (the default)

9. Click Save

10. Back at the original Choose Server Configuration menu, click Connect to start listening for a connection fromIceberg

Note: This method requires that Iceberg can connect to port 11111 on your machine. You may need to modify yourmachine’s firewall to permit such connections. On Linux machines using the UFW firewall you can allow connectionsto port 11111 on your machine from other machines on the University network (including Iceberg) and from no othermachines using:

sudo ufw allow from 143.167.0.0/16 to any port 11111

Start the Server

After configuring the client:

1 It is not possible to install the latest version of the paraview GUI on iceberg due to the Qt version shipped with Scientific Linux 5.2 The client and server versions have to match.


http://paraview.org/

http://www.paraview.org/documentation/

http://www.paraview.org/

http://www.paraview.org/download/

https://wiki.archlinux.org/index.php/Uncomplicated_Firewall


1. Log in to iceberg from the client machine via ssh3

2. Run qsub-paraview from this login node (not from a worker node)

This will submit a job to the scheduler queue for 16 processes with 4GB of RAM each. This is designed to be used forlarge visualisation tasks, smaller jobs can be requested by specifying standard qsub commands to qsub-paraview.For example, to request just one process:

qsub-paraview -pe openmpi-ib 1

Assuming you still have the client listening for connections, once the paraview job starts in the queue it should connectto your client and you should be able to start accessing data stored on iceberg and rendering images.

A Note on Performance

When you run Paraview locally it will use the graphics hardware of your local machine for rendering. This is usinghardware in your computer to create and display these images. On iceberg there is no such hardware, it is simulatedvia software. Therefore for small datasets you will probably find you are paying a performance penalty for usingParaview on iceberg.

The advantage however, is that the renderer is on the same cluster as your data, so no data transfer is needed, and youhave access to very large amounts of memory for visualising very large datasets.

Manually Starting the Server

The qsub-paraview command is a wrapper that automatically detects the client IP address from the SSH con-nection and submits the job. It is possible to customise this behavior by copying and modifying this script. This forinstance would allow you to start the paraview server via MyApps or from a different computer to the one with theclient installed. The script used by qsub-paraview also serves as a good example script and can be copied intoyour home directory by running cp /usr/local/bin/pvserver_submit.sh ~/. This script can then beqsubmitted as normal by qsub. The client IP address can be added manually by replacing echo $SSH_CLIENT |awk '{ print $1}' with the IP address. More information on Paraview client/server can be found here.

Installation

Custom build scripts are availible in /usr/local/extras/paraview/build_scripts which can be usedto recompile.

Perl

Perl

Latest Version 5.24.0

URL https://www.perl.org/

Perl 5 is a programming language originally designed for text/report processing but now widely used by the bioin-formatics community due to its ability to identify and process patterns in sequences of data. Perl 5 is also used byLinux/UNIX systems administrators as it has borrowed much from shell scripting languages and contains functionalityfor manipulating files and directory trees.

3 Connecting to Paraview via the automatic method descibed here is not supported on the MyApps portal.

2.2. Iceberg 137

http://www.paraview.org/Wiki/Setting_up_a_ParaView_Server#Running_the_Server

https://www.perl.org/


Usage

The default version of Perl on the system is 5.10.1; no module command is required to use it

$ perl --version

This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

However you may see warnings when trying to install Perl software that this version (or the cran package managetool that comes with it) is too old; in this case you can activate a more recent version using

module load apps/gcc/4.4.7/perl/5.24.0

Several Perl modules have been upgraded/installed for this instance of Perl:

• CPAN, the original Perl package management tool, was upgraded to 2.14

• BioPerl 1.007000 (aka v1.7.0) was installed. BioPerl is a collection of Perl modules that facilitate the develop-ment of Perl scripts for bioinformatics applications. It has played an integral role in the Human Genome Project.BioPerl has pre-requisites of inc::latest (0.500) and Module::Build (0.4220).

• Term::ReadKey (2.37) and Term::Readline::Gnu (1.34) were installed; these make interactive use ofCPAN more pleasant.

• local::lib was installed, which allows users to install additional Perl modules in their home directories.MORE INFO NEEDED (see Installing Perl modules).

• App::cpanminus (1.7042) was installed, which provides the cpanm program. This is considered by someto be a superior package management tool over cpan.

You can confirm that you are using this newer version of Perl (and have access to BioPerl) using

$ perl --version

Summary of my perl5 (revision 5 version 24 subversion 0) configuration:

$ perl -MBio::Root::Version -le 'print $Bio::Root::Version::VERSION'

1.007000...

Performance: 5.24.0 can be significantly faster than earlier versions of Perl 5 for arithemetic operations.

Installing Perl modules

If using a version of Perl activated using module load then you can subsequently easily install Perl modules inyour home directory (within what’s called a Perl local::lib). Run the following

eval $(perlvers="$(perl -e 'print $^V')"; perl -I$HOME/.perl$perlvers/lib/perl5 -→˓Mlocal::lib=$HOME/.perl$perlvers)

The next time you try to use cpanm to install a package it should install it into the (hidden) directory .perl5.x.ywithin your home directory. For example, the following installs the Bio::Tradis Perl module into ~/.perl5.24.0/


http://bioperl.org/

https://www.foo.be/docs/tpj/issues/vol1_2/tpj0102-0001.html

https://www.nu42.com/2015/12/go-faster-stripes-for-perl-programs.html


module load apps/gcc/4.4.7/perl/5.24.0eval $(perlvers="$(perl -e 'print $^V')"; perl -I$HOME/.perl$perlvers/lib/perl5 -→˓Mlocal::lib=$HOME/.perl$perlvers)cpanm Bio::Tradis

If you always want to have your Perl local::lib available then you may want to add the eval ... line above to your~/.bashrc file.

Installation instructions

Version 5.24.0

1. Run this script to download, unpack, build (using the system GCC (4.4.7)), test then install Perl (in /usr/local/packages6/apps/gcc/4.4.7/perl/5.24.0/)

2. Install download:this modulefile </iceberg/software/modulefiles/apps/gcc/4.4.7/perl/5.24.0> as /usr/local/modulefiles/apps/gcc/4.4.7/perl/5.24.0/.

PerlBrew could have been used to install and manage multiple versions of Perl but it offers relatively few advantagesover Environment Modules and the latter are already used system-wide for package management.

Phyluce

Phyluce

Version 1.5.0

Dependancies apps/python/conda

URL https://github.com/faircloth-lab/phyluce

Documentation http://phyluce.readthedocs.io/

Phyluce (phy-loo-chee) is a software package that was initially developed for analyzing data collected from ultracon-served elements in organismal genomes.

Usage

Phyluce can be activated using the module file:

module load apps/binapps/phyluce/1.5.0

Phyluce makes use of the apps/python/conda module, therefore this module will be loaded by loading Phyluce.As Phyluce is a Python package your default Python interpreter will be changed by loading Phyluce.

Installation notes

As root:

2.2. Iceberg 139

https://perlbrew.pl/

https://github.com/faircloth-lab/phyluce

http://phyluce.readthedocs.io/


$ module load apps/python/conda$ conda create -p /usr/local/packages6/apps/binapps/conda/phyluce python=2$ source activate /usr/local/packages6/apps/binapps/conda/phyluce$ conda install -c https://conda.binstar.org/faircloth-lab phyluce

This installs Phyluce as a conda environment in the /usr/local/packages6/apps/binapps/conda/phyluce folder, which is then loaded by the module file phyluce/1.5.0, which is a modification of the anacondamodule files.

Picard

Picard

Version 1.129

URL https://github.com/broadinstitute/picard/

A set of Java command line tools for manipulating high-throughput sequencing (HTS) data and formats. Picard isimplemented using the HTSJDK Java libraryHTSJDK, supporting accessing of common file formats, such as SAMand VCF, used for high-throughput sequencing data.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx or qrshcommand, preferably requesting at least 8 Gigabytes of memory

qrsh -l rmem=8G

The latest version of Picard (currently 1.129) is made available with the command

module load apps/binapps/picard


module load apps/binapps/picard/1.129module load apps/binapps/picard/1.101

These module commands also changes the environment to use Java 1.6 since this is required by Picard 1.129. Anenvironment variable called PICARDHOME is created by the module command that contains the path to the requestedversion of Picard.

Thus, you can run the program with the command

java -jar $PICARDHOME/picard.jar

Installation notes

Version 1.129

A binary install was used. The binary came from the releases page of the project’s github repo


https://github.com/broadinstitute/picard/


unzip picard-tools-1.129.zipmkdir -p /usr/local/packages6/apps/binapps/picardmv ./picard-tools-1.129 /usr/local/packages6/apps/binapps/picard/1.129

Version 1.101

A binary install was used. The binary came from the project’s sourceforge site https://sourceforge.net/projects/picard/files/picard-tools/

unzip picard-tools-1.101.zipmv ./picard-tools-1.101 /usr/local/packages6/apps/binapps/picard/1.101/

Modulefile

Version 1.129

The module file is on the system at /usr/local/modulefiles/apps/binapps/picard/1.129

Its contents are

#%Module1.0######################################################################### Picard 1.129 modulefile##


#This version of Picard needs Java 1.6module load apps/java/1.6

proc ModulesHelp { } {puts stderr "Makes Picard 1.129 available"

}

set version 1.129set PICARD_DIR /usr/local/packages6/apps/binapps/picard/$version

module-whatis "Makes Picard 1.129 available"

prepend-path PICARDHOME $PICARD_DIR

Version 1.101

The module file is on the system at /usr/local/modulefiles/apps/binapps/picard/1.101

Its contents are

#%Module1.0######################################################################### Picard 1.101 modulefile##


2.2. Iceberg 141

https://sourceforge.net/projects/picard/files/picard-tools/

https://sourceforge.net/projects/picard/files/picard-tools/


#This version of Picard needs Java 1.6module load apps/java/1.6

proc ModulesHelp { } {puts stderr "Makes Picard 1.101 available"

}

set version 1.101set PICARD_DIR /usr/local/packages6/apps/binapps/picard/$version

module-whatis "Makes Picard 1.101 available"

prepend-path PICARDHOME $PICARD_DIR

Plink

Plink

Versions 1.90 beta 3.42


Dependancies None

URL https://www.cog-genomics.org/plink2

PLINK is a free, open-source whole genome association analysis toolset, designed to perform a range of basic, large-scale analyses in a computationally efficient manner.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive sesssion with the `qsh or qrshcommand.

The latest version of Plink is made available with the command

module load apps/binapps/plink

Alternatively, you can load a specific version. To access the latest version (build date 20 Sep 2016)

module load apps/binapps/plink/1.90b3.42

An older build of Plink 1.9 (15 Jul 2015) is also available

module load apps/binapps/plink/1.90b3v

After making a version of Plink available you can then run it using plink on the command line.

Installation notes



https://www.cog-genomics.org/plink2


Both versions of Plink were installed like so

$ver=1.90b3.42 # or 1.90b3v

mkdir plink_buildcd plink_buildunzip plink_linux_x86_64.ziprm plink_linux_x86_64.zipmkdir -p /usr/local/packages6/apps/binapps/plink/$vermv * /usr/local/packages6/apps/binapps/plink/$ver

The modulefiles are at /usr/local/modulefiles/apps/binapps/plink/$ver

#%Module10.2#####################################################################



puts stderr " Makes Plink $ver available to the system."}

# Plink version (not in the user's environment)set ver 1.90b3.42 # or 1.90b3v

module-whatis "sets the necessary Plink $ver paths"

prepend-path PATH /usr/local/packages6/apps/binapps/plink/$ver

povray

povray

Version 3.7.0

URL http://www.povray.org/

The Persistence of Vision Raytracer is a high-quality, Free Software tool for creating stunning three-dimensionalgraphics. The source code is available for those wanting to do their own ports.

Interactive Usage

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. The latest version of povray (currently 3.7) is made available with the command

module load apps/gcc/5.2/povray


2.2. Iceberg 143

http://www.povray.org/


module load apps/gcc/5.2/povray/3.7

This command makes the povray binary available to your session. It also loads version 5.2 of the gcc compilerenvironment since gcc 5.2 was used to compile povray 3.7.

You can now run povray. For example, to confirm the version loaded

povray --version

and to get help

povray --help

Documentation

Once you have made povray available to the system using the module command above, you can read the man pagesby typing

man povray

Installation notes

povray 3.7.0 was installed using gcc 5.2 using the following script

• install_povray-3.7.sh

Testing

The test suite was executed

make check

All tests passed.

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/5.2/povray/3.7


Python

Python

Support Level Gold

Dependencies None

URL https://python.org


https://github.com/rcgsheffield/sheffield_hpc/blob/master/iceberg/software/install_scripts/apps/gcc/5.2/povray/0.37/install_povray-3.7.sh

https://python.org


Version All

This page documents the “miniconda” installation on iceberg. This is the recommended way of using Python oniceberg, and the best way to be able to configure custom sets of packages for your use.

“conda” a Python package manager, allows you to create “environments” which are sets of packages that you canmodify. It does this by installing them in your home area. This page will guide you through loading conda and thencreating and modifying environments so you can install and use whatever Python packages you need.

Using conda Python

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand.

Conda Python can be loaded with:

module load apps/python/conda

The root conda environment (the default) provides Python 3 and no extra modules, it is automatically updated, andnot recommended for general use, just as a base for your own environments. There is also a python2 environment,which is the same but with a Python 2 installation.

Quickly Loading Anaconda Environments

There are a small number of environments provided for everyone to use, these are the default root and python2environments as well as various versions of Anaconda for Python 3 and Python 2.

The anaconda environments can be loaded through provided module files:

module load apps/python/anaconda2-2.4.0module load apps/python/anaconda3-2.4.0module load apps/python/anaconda3-2.5.0

Where anaconda2 represents Python 2 installations and anaconda3 represents Python 3 installations. Thesecommands will also load the apps/python/conda module and then activate the anaconda environment specified.

Note: Anaconda 2.5.0 is compiled with Intel MKL libraries which should result in higher numerical performance.

Using conda Environments

Once the conda module is loaded you have to load or create the desired conda environments. For the documentationon conda environments see the conda documentation.

You can load a conda environment with:

source activate python2

where python2 is the name of the environment, and unload one with:

source deactivate

2.2. Iceberg 145

http://conda.pydata.org/docs/using/envs.html


which will return you to the root environment.

It is possible to list all the available environments with:

conda env list

Provided system-wide are a set of anaconda environments, these will be installed with the anaconda version numberin the environment name, and never modified. They will therefore provide a static base for derivative environments orfor using directly.

Creating an Environment

Every user can create their own environments, and packages shared with the system-wide environments will not bereinstalled or copied to your file store, they will be symlinked, this reduces the space you need in your /homedirectory to install many different Python environments.

To create a clean environment with just Python 2 and numpy you can run:

conda create -n mynumpy python=2.7 numpy

This will download the latest release of Python 2.7 and numpy, and create an environment named mynumpy.

Any version of Python or list of packages can be provided:

conda create -n myscience python=3.5 numpy=1.8.1 scipy

If you wish to modify an existing environment, such as one of the anaconda installations, you can clone that envi-ronment:

conda create --clone anaconda3-2.3.0 -n myexperiment

This will create an environment called myexperiment which has all the anaconda 2.3.0 packages installed withPython 3.

Installing Packages Inside an Environment

Once you have created your own environment you can install additional packages or different versions of packagesinto it. There are two methods for doing this, conda and pip, if a package is available through conda it is stronglyrecommended that you use conda to install packages. You can search for packages using conda:

conda search pandas

then install the package using:

conda install pandas

if you are not in your environment you will get a permission denied error when trying to install packages, if thishappens, create or activate an environment you own.

If a package is not available through conda you can search for and install it using pip:

pip search colormath

pip install colormath



Previous Anaconda Installation

There is a legacy anaconda installation which is accessible through the binapps/anacondapython/2.3module.This module should be considered deprecated and should no longer be used.

Using Python with MPI

There is an experimental set of packages for conda that have been compiled by the iceberg team, which allow youto use a MPI stack entirely managed by conda. This allows you to easily create complex evironments and use MPIwithout worrying about other modules or system libraries.

To get access to these packages you need to run the following command to add the repo to your conda config:

conda config --add channels file:///usr/local/packages6/conda/conda-bld/

you should then be able to install the packages with the openmpi feature, which currently include openmpi, hdf5,mpi4py and h5py:

conda create -n mpi python=3.5 openmpi mpi4py

Currently, there are Python 2.7, 3.4 and 3.5 versions of mpi4py and h5py compiled in this repository.

The build scripts for these packages can be found in this GitHub repository.

Installation Notes


The conda package manager is installed in /usr/share/packages6/conda, it was installed using the minicondainstaller.

The two “root” environments root and python2 can be updated using the update script located in /usr/local/packages6/conda/_envronments/conda-autoupdate.sh. This should be run regularly to keep thisbase environments upto date with Python, and more importantly with the conda package manager itself.

Installing a New Version of Anaconda

Perform the following:

$ cd /usr/local/packages6/conda/_envronments/$ cp anaconda2-2.3.0.yml anaconda2-x.y.z.yml

then edit that file modifying the environment name and the anaconda version under requirements then run:

$ conda env create -f anaconda2-x.y.z.yml

then repeat for the Python 3 installation.

Then copy the modulefile for the previous version of anaconda to the new version and update the name of the environ-ment. Also you will need to append the new module to the conflict line in apps/python/.conda-environments.tcl.

Modulefiles

• The conda modulefile was installed as /usr/local/modulefiles/apps/python/conda.

2.2. Iceberg 147

https://github.com/rcgsheffield/conda-packages

http://conda.pydata.org/miniconda.html


R

R

Dependencies BLAS



R is a statistical computing language.

Interactive Usage


The latest version of R can be loaded with

module load apps/R

Alternatively, you can load a specific version of R using one of the following

module load apps/R/3.3.1module load apps/R/3.3.0module load apps/R/3.2.4module load apps/R/3.2.3module load apps/R/3.2.2module load apps/R/3.2.1module load apps/R/3.2.0module load apps/R/3.1.2

R can then be run with

$ R


Here, we assume that you wish to run the program my_code.R on the system. With batch usage it is recommendedto load a specific version of R, for example module load apps/R/3.2.4, to ensure the expected output isachieved.

First, you need to write a batch submission file. We assume you’ll call this my_job.sge

#!/bin/bash#$ -cwd # Run job from current directory#$ -l rmem=4G # Request 4 gigabytes of memory

module load apps/R/3.3.1 # Recommended to load a specific version of R

R CMD BATCH my_code.r my_code.r.o$JOB_ID

Note that R must be called with both the CMD and BATCH options which tell it to run an R program, in this casemy_code.R. If you do not do this, R will attempt to open an interactive prompt.





The final argument, my_code.R.o$JOBID, tells R to send output to a file with this name. Since $JOBID willalways be unique, this ensures that all of your output files are unique. Without this argument R sends all output to afile called my_code.Rout.

Ensuring that my_code.R and my_job.sge are both in your current working directory, submit your job to thebatch system

qsub my_job.sge

Replace my_job.sge with the name of your submission script.

Graphical output

By default, graphical output from batch jobs is sent to a file called Rplots.pdf


As you will not have permissions to install packages to the default folder, additional R packages can be installed toyour home folder ~/. To create the appropriate folder, install your first package in R in interactive mode. Load aninteractive R session as described above, and install a package with:

install.packages()

You will be prompted to create a personal package library. Choose yes. The package will download and install from aCRAN mirror (you may be asked to select a nearby mirror, which you can do simply by entering the number of yourpreferred mirror).

Once the chosen package has been installed, additional packages can be installed either in the same way, or by creatinga .R script. An example script might look like

install.packages("dplyr")install.packages("devtools")

Call this using source(). For example if your script is called packages.R and is stored in your home folder,source this from an interactive R session with

source("~/packages.R")

These additional packages will be installed without prompting to your personal package library.

To check your packages are up to date, and update them if necessary, run the following line from an R interactivesession

update.packages(lib.loc = "~/R/x86_64-unknown-linux-gnu-library/3.3/")

The folder name after ~/R/ will likely change, but this can be completed with tab autocompletion from the R session.Ensure lib.loc folder is specified, or R will attempt to update the wrong library.

Warning: Notice that the personal package library path includes the version of R: if after installing some packagesyou switch to using a different major or minor version of R then you will need then to install those package for thisnew version.

2.2. Iceberg 149

http://semver.org/


R Packages that require external libraries

Some R packages require external libraries to be installed before you can install and use them. Since there are somany, we only install those libraries that have been explicitly requested by users of the system.

The associated R packages are not included in the system install of R, so you will need to install them yourself to yourhome directory following the instructions linked to below.

• geos This is the library required for the rgeos package.

• Geospatial Data Abstraction Library (GDAL) and PROJ.4 These are the libraries required for the rgdal pack-age.

• JAGS This is the library required for the rjags and runjags packages

Warning: To install r packages that require external libraries, the libraries need to be loaded prior to installingthe r packages. E.g. to install package rgeos you would need to load geos, enter an interactive R session and theninstall rgeos

module load libs/geos/3.6.1/gcc-4.9.4Rinstall.packages("rgeos")

See here more information on the available external libraries

Using the Rmath library in C Programs

The Rmath library allows you to access some of R’s functionality from a C program. For example, consider theC-program below

#include <stdio.h>#define MATHLIB_STANDALONE#include "Rmath.h"

main(){double shape1,shape2,prob;

shape1 = 1.0;shape2 = 2.0;prob = 0.5;

printf("Critical value is %lf\n",qbeta(prob,shape1,shape2,1,0));}

This makes use of R’s qbeta function. You can compile and run this on a worker node as follows.

Start a session on a worker node with qrsh or qsh and load the R module

module load apps/R/3.3.0

Assuming the program is called test_rmath.c, compile with

gcc test_rmath.c -lRmath -lm -o test_rmath

For full details about the functions made available by the Rmath library, see section 6.7 of the document Writing Rextensions


https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Numerical-analysis-subroutines



Accelerated version of R

There is an experimental, accelerated version of R installed on Iceberg that makes use of the Intel compilers and theIntel MKL. See R (Intel Build) for details.

Installation Notes


version 3.3.1

• What’s new in R version 3.3.1

Compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled. This build required several externalmodules including xz utils, curl, bzip2 and zlib

• Installation script (install.sh): downloads, compiles, tests and installs R 3.3.1 and the Rmathlibrary.

– Install log files, including the output of the make check tests are available on the system at /usr/local/packages6/R/3.3.1/install_logs

• module file installed as /usr/local/modulefiles/apps/R/3.3.1

version 3.3.0


This was a scripted install. It was compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled. Youwill need a large memory qrshx session in order to successfully run the build script. I used qrshx -l rmem=8G-l mem=8G

This build required several external modules including xz utils, curl, bzip2 and zlib

• install_R_3.3.0.sh Downloads, compiles, tests and installs R 3.3.0 and the Rmath library.

• R 3.3.0 Modulefile located on the system at /usr/local/modulefiles/apps/R/3.3.0

• Install log-files, including the output of the make check tests are available on the system at /usr/local/packages6/R/3.3.0/install_logs

Version 3.2.4


This was a scripted install. It was compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled. Youwill need a large memory qrshx session in order to successfully run the build script. I used qrshx -l rmem=8G-l mem=8G

This build made use of new versions of xz utils and curl




2.2. Iceberg 151

https://stat.ethz.ch/pipermail/r-announce/2016/000604.html


https://cran.r-project.org/bin/windows/base/old/3.2.4/NEWS.R-3.2.4.html


Version 3.2.3


This was a scripted install. It was compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled. Youwill need a large memory qrsh session in order to successfully run the build script. I used qrsh -l rmem=8G -lmem=16G




Version 3.2.2


This was a scripted install. It was compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled. Youwill need a large memory qrsh session in order to successfully run the build script. I used qrsh -l rmem=8G -lmem=16G

• install_R_3.2.2.sh Downloads, compiles and installs R 3.2.2 and the Rmath library.


• Install log-files were manually copied to /usr/local/packages6/R/3.2.2/install_logs on thesystem. This step should be included in the next version of the install script.

Version 3.2.1

This was a manual install. It was compiled from source with gcc 4.4.7 and with --enable-R-shlib enabled.

• Install notes


Older versions

Install notes for older versions of R are not available.

relion

relion

Versions 1.4

URL http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page

RELION is a software package that performs an empirical Bayesian approach to (cryo-EM) structure de-terminationby single-particle analysis. Note that RELION is distributed under a GPL license.


https://cran.r-project.org/bin/windows/base/old/3.2.3/NEWS.R-3.2.3.html


http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page


Making RELION available

The following module command makes the latest version of gemini available to your session

module load apps/gcc/4.4.7/relion


module load apps/gcc/4.4.7/relion/1.4

Installation notes


Install instructions: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Download_%26_install

• RELION was installed using the gcc 4.4.7 compiler and Openmpi 1.8.8

• install_relion.sh

• Note - the environment variable RELION_QSUB_TEMPLATE points to an SGE qsub template, which needscustomizing to work with our environment

Modulefile

The module file is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/relion/1.4


#%Module1.0######################################################################### relion module file##



puts stderr " Setups `relion-$relionversion' environment variables"}

set relionversion 1.4

module load apps/gcc/4.4.7/ctffind/3.140609module load apps/binapps/resmap/1.1.4module load mpi/gcc/openmpi/1.8.8

prepend-path PATH /usr/local/packages6/apps/gcc/4.4.7/relion/1.4/binprepend-path LD_LIBRARY_PATH /usr/local/packages6/apps/gcc/4.4.7/relion/1.4/libsetenv RELION_QSUB_TEMPLATE /usr/local/packages6/apps/gcc/4.4.7/relion/1.4/bin/relion_→˓qsub.cshsetenv RELION_CTFFIND_EXECUTABLE ctffind3_mp.exesetenv RELION_RESMAP_EXECUTABLE /usr/local/packages6/apps/binapps/resmap/1.1.4

2.2. Iceberg 153

http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Download_%26_install


ResMap

ResMap

Versions 1.1.4

URL http://resmap.sourceforge.net/

ResMap (Resolution Map) is a software package for computing the local resolution of 3D density maps studied instructural biology, primarily electron cryo-microscopy.

Making ResMap available

The following module command makes the latest version of gemini available to your session

module load apps/binapps/resmap


module load apps/binapps/resmap/1.1.4

Installation notes


• ResMap was installed as a precompiled binary from https://sourceforge.net/projects/resmap/files/ResMap-1.1.4-linux64/download

Modulefile

The module file is on the system at /usr/local/modulefiles/apps/binapps/1.1.4


#%Module1.0######################################################################### ResMap module file##



puts stderr " Adds `ResMap-$resmapversion' to your PATH environment variable"}

set resmapversion 1.1.4

prepend-path PATH /usr/local/packages6/apps/binapps/resmap/$resmapversion/


http://resmap.sourceforge.net/

https://sourceforge.net/projects/resmap/files/ResMap-1.1.4-linux64/download

https://sourceforge.net/projects/resmap/files/ResMap-1.1.4-linux64/download


Samtools

Samtools

Versions 1.3.1 1.2

URL http://samtools.sourceforge.net/

SAM (Sequence Alignment/Map) format is a generic format for storing large nucleotide sequence alignments.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qsh command.

Next, load a specific version of Samtools with one of the following:

module load apps/gcc/6.2/samtools/1.3.1module load apps/gcc/5.2/samtools/1.2

This command makes the Samtools binary directory available to your session.

Documentation

Once you have made Samtools available to the system using the module command above, you can read the manpages by typing

man samtools

Installation notes

This section is primarily for system administrators.

Version 1.3.1

This install script:

1. Built Samtools plus the bundled HTSlib, HTSlib utilities such as bgzip plus various useful plugins. Compiledusing GCC 6.2 (GNU Compiler Collection (gcc)).

2. Ran all tests using make tests; a summary of the results is shown below; for full results see /usr/local/packages6/apps/gcc/6.2/samtools/1.3.1/tests.log

Number of tests:total .. 423passed .. 403failed .. 0expected failure .. 20unexpected pass .. 0

test/merge/test_bam_translate test/merge/test_bam_translate.tmptest/merge/test_rtrans_buildtest/merge/test_trans_tbl_initcd test/mpileup && ./regression.sh mpileup.reg

2.2. Iceberg 155

http://samtools.sourceforge.net/


=== Testing mpileup.reg regressions ===

Expected passes: 124Unexpected passes: 0Expected failures: 0Unexpected failures: 0=> PASScd test/mpileup && ./regression.sh depth.reg

=== Testing depth.reg regressions ===

Expected passes: 14Unexpected passes: 0Expected failures: 0Unexpected failures: 0=> PASS

3. Installed Samtools to /usr/local/packages6/apps/gcc/6.2/samtools/1.3.1

Next, this modulefile was installed as /usr/local/modulefiles/apps/gcc/6.2/samtools/1.3.1

Version 1.2

Installed using GCC 5.2 (GNU Compiler Collection (gcc))


tar -xvjf ./samtools-1.2.tar.bz2cd samtools-1.2mkdir -p /usr/local/packages6/apps/gcc/5.2/samtools/1.2make prefix=/usr/local/packages6/apps/gcc/5.2/samtools/1.2make prefix=/usr/local/packages6/apps/gcc/5.2/samtools/1.2 install#tabix and bgzip are not installed by the above procedure.#We can get them by doing the followingcd htslib-1.2.1/makemv ./tabix /usr/local/packages6/apps/gcc/5.2/samtools/1.2/bin/mv ./bgzip /usr/local/packages6/apps/gcc/5.2/samtools/1.2/bin/

The test suite was run with

make test 2>&1 | tee make_tests.log

The summary of the test output was

Test output:Number of tests:

total .. 368passed .. 336failed .. 0expected failure .. 32unexpected pass .. 0

test/merge/test_bam_translate test/merge/test_bam_translate.tmptest/merge/test_pretty_headertest/merge/test_rtrans_build



test/merge/test_trans_tbl_initcd test/mpileup && ./regression.shSamtools mpileup tests:

EXPECTED FAIL: Task failed, but expected to fail;when running $samtools mpileup -x -d 8500 -B -f mpileup.ref.fa deep.sam|awk '{print→˓$4}'

Expected passes: 123Unexpected passes: 0Expected failures: 1Unexpected failures: 0

The full log is on the system at /usr/local/packages6/apps/gcc/5.2/samtools/1.2/make_tests.log

This modulefile was installed as /usr/local/modulefiles/apps/gcc/5.2/samtools/1.2

sbt

sbt

Versions 0.13.13

URL http://www.scala-sbt.org/

sbt is a build tool for Scala, Java, and more.

Interactive Usage


The latest version of sbt (currently 0.13.13) is made available with the command

module load apps/binapps/sbt


module load apps/binapps/sbt/0.13.13

This command makes the sbt command available to your session.

Installation notes


sbt 0.13.13 was installed as follows

tar -xvzf ./sbt-0.13.13.tgzcd sbt-launcher-packaging-0.13.13/mkdir -p /usr/local/packages6/apps/binapps/scala-sbt/0.13.13mv ./* /usr/local/packages6/apps/binapps/scala-sbt/0.13.13/

2.2. Iceberg 157

http://www.scala-sbt.org/


Module files

• The module file is on the system at /usr/local/modulefiles/apps/binapps/sbt

• On github: 0.13.13.

spark

Spark

Version 2.0

URL http://spark.apache.org/

Apache Spark is a fast and general engine for large-scale data processing.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrsh or qrshxcommand.

To make Spark available, execute the following module command:

module load apps/gcc/4.4.7/spark/2.0

Installation notes

Spark was built using the system gcc 4.4.7:

tar -xvzf ./spark-2.0.0.tgzcd spark-2.0.0build/mvn -DskipTests clean package

mkdir -p /usr/local/packages6/apps/gcc/4.4.7/sparkmv spark-2.0.0 /usr/local/packages6/apps/gcc/4.4.7/spark/

The default install of Spark is incredibly verbose. Even a ‘Hello World’ program results in many lines of [INFO]. Tomake it a little quieter, reduce the default log4j level from INFO to WARN:

cd /usr/local/packages6/apps/gcc/4.4.7/spark/spark-2.0.0/conf/cp log4j.properties.template log4j.properties

Edit the file log4j.properties so that the line beginning log4j.rootCategory reads:

log4j.rootCategory=WARN, console

Modulefile

Version 2.0

The module file is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/spark/2.0


http://spark.apache.org/


Its contents are

#%Module1.0######################################################################### Spark module file##


set sparkhome /usr/local/packages6/apps/gcc/4.4.7/spark/spark-2.0.0

# Use only one core. User can override this if they wantsetenv MASTER local\[1\]setenv SPARK_HOME $sparkhomeprepend-path PATH $sparkhome/bin

SU2

SU2

Version 4.1.0

Dependancies gcc/4.4.7 mpi/gcc/openmpi/1.10.1

URL http://su2.stanford.edu/

Documentation https://github.com/su2code/SU2/wiki

The SU2 suite is an open-source collection of C++ based software tools for performing Partial Differential Equation(PDE) analysis and solving PDE constrained optimization problems.

Usage

SU2 can be activated using the module file:

module load apps/gcc/4.4.7/su2

Installation notes

Su2 was compiled using the install_su2.sh script.

SU2 also has it’s own version of CGNS which was compiled with the script install_cgns.sh.

The module file is 4.1.0.

Theano

Theano

2.2. Iceberg 159

http://su2.stanford.edu/

https://github.com/su2code/SU2/wiki


URL http://deeplearning.net/software/theano/index.html

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involvingmulti-dimensional arrays efficiently. Theano is most commonly used to perform deep learning and has excellent GPUsupport and integration through PyCUDA. The following steps can be used to setup and configure Theano on yourown profile.

About Theano on Iceberg

As Theano and all its dependencies are written in Python, it can be installed locally in your home directory. The useof Anaconda (Python) is recommended as it is able to create a virtual environment in your home directory, allowingthe installation of new Python packages without admin permission.

Installation

First request an interactive session, e.g. with qrshx. To use GPUs see Interactive use of the GPUs.

Load the relevant modules with the following command:

module load apps/python/anaconda3-2.5.0module load libs/binlibs/cudnn/5.1-cuda-8.0.44

Create a conda environment to load relevant modules on your local user account

conda create -n theano python=3.5source activate theano

Install the other Python module dependencies which are required using pip (alternatively these could be installed withconda if you prefer)

pip install theano nose nose-parameterized pycuda

For optimal Theano performance, enable the CUDA memory manager CNMeM. To do this, create the .theanorc file inyour HOME directory and set the fraction of GPU memory reserved by Theano. The exact amount of energy may haveto be hand-picked: if Theano asks for more memory that is currently available on the GPU, an error will be thrownduring import of theano module. Create or edit the .theanorc file with nano

nano ~/.theanorc

Add the following lines and, if necessary, change the 0.8 number to whatever works for you

[lib]cnmem=0.8

Run python and verify that Theano is working correctly

python -c "import theano;theano.test()"

Every Session Afterwards and in Your Job Scripts

The previous instuctions installs Theano and its dependencies inside your home directory but every time you use a newsession or within your job scripts, the modules must be loaded and conda must be activated again. Use the followingcommand to activate the Conda environment with Theano installed:


http://deeplearning.net/software/theano/index.html


module load apps/python/anaconda3-2.5.0module load libs/binlibs/cudnn/5.1-cuda-8.0.44source activate theano

Tophat

Tophat

Versions 2.1.0

URL https://ccb.jhu.edu/software/tophat/index.shtml

TopHat is a fast splice junction mapper for RNA-Seq reads. It aligns RNA-Seq reads to mammalian-sized genomesusing the ultra high-throughput short read aligner Bowtie, and then analyzes the mapping results to identify splicejunctions between exons.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with either the qsh or qrshcommand.

The latest version of tophat (currently 2.1.0) is made available with the command:

module apps/gcc/4.8.2/tophat


module load apps/gcc/4.8.2/tophat/2.1.0

This command makes the tophat binary available to your session.

Installation notes

Tophat 2.1.0 was installed using gcc 4.8.2. Installs were attempted using gcc 5.2 and gcc 4.4.7 but both failed (see thisissue on github )

This install has dependencies on the following

• GNU Compiler Collection (gcc) 4.8.2

• Boost C++ Library 1.58

• Bowtie2 (not needed at install time but is needed at runtime)

Install details

module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/boost/1.58

mkdir -p /usr/local/packages6/apps/gcc/4.8.2/tophat/2.1.0

tar -xvzf ./tophat-2.1.0.tar.gzcd tophat-2.1.0./configure --with-boost=/usr/local/packages6/libs/gcc/4.8.2/boost/1.58.0/ --prefix=/→˓usr/local/packages6/apps/gcc/4.8.2/tophat/2.1.0

2.2. Iceberg 161

https://ccb.jhu.edu/software/tophat/index.shtml




Configuration results

-- tophat 2.1.0 Configuration Results --C++ compiler: g++ -Wall -Wno-strict-aliasing -g -gdwarf-2 -Wuninitialized -O3→˓ -DNDEBUG -I./samtools-0.1.18 -pthread -I/usr/local/packages6/libs/gcc/4.8.2/boost/→˓1.58.0//include -I./SeqAn-1.3Linker flags: -L./samtools-0.1.18 -L/usr/local/packages6/libs/gcc/4.8.2/boost/→˓1.58.0//libBOOST libraries: -lboost_thread -lboost_systemGCC version: gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)Host System type: x86_64-unknown-linux-gnuInstall prefix: /usr/local/packages6/apps/gcc/4.8.2/tophat/2.1.0Install eprefix: ${prefix}

Built with

makemake install

Testing

A test script was executed based on the documentation on the tophat website (retrieved 29th October 2015). It onlyproves that the code can run without error, not that the result is correct.

• tophat_test.sh

Modulefile

• The module file is on the system at /usr/local/modulefiles/apps/gcc/4.8.2/tophat/2.1.0

• The module file is on github

VCFtools

VCFtools

Version 0.1.14

Dependancies gcc/5.2

URL https://vcftools.github.io/

Documentation https://vcftools.github.io/examples.html

VCFtools is a program package designed for working with VCF files, such as those generated by the 1000 GenomesProject. The aim of VCFtools is to provide easily accessible methods for working with complex genetic variation datain the form of VCF files.

Usage

VCFtools can be activated using the module file:


https://ccb.jhu.edu/software/tophat/tutorial.shtml

https://vcftools.github.io/

https://vcftools.github.io/examples.html


module load apps/gcc/5.2/vcftools/0.1.14

This will both add the binary files to the $PATH and the Perl module to $PERL5LIB.

Installation notes

VCFtools was compiled using the install_vcftools.sh script, the module file is 0.1.14.

Velvet

Velvet

Version 1.2.10

URL https://www.ebi.ac.uk/~zerbino/velvet/

Sequence assembler for very short reads.

Interactive Usage

After connecting to Iceberg (see Establishing a SSH connection), start an interactive session with the qrshx or qrshcommand.

To add the velvet binaries to the system PATH, execute the following command

module load apps/gcc/4.4.7/velvet/1.2.10

This makes two programs available:-

• velvetg - de Bruijn graph construction, error removal and repeat resolution

• velveth - simple hashing program

Example submission script

If the command you want to run is velvetg /fastdata/foo1bar/velvet/assembly_31 -exp_covauto -cov_cutoff auto, here is an example submission script that requests 60 GB of memory

#!/bin/bash#$ -l rmem=60G

module load apps/gcc/4.4.7/velvet/1.2.10

velvetg /fastdata/foo1bar/velvet/assembly_31 -exp_cov auto -cov_cutoff auto

Put the above into a text file called submit_velvet.sh and submit it to the queue with the command qsubsubmit_velvet.sh

2.2. Iceberg 163

https://www.ebi.ac.uk/~zerbino/velvet/


Velvet Performance

Velvet has got multicore capabilities but these have not been compiled in our version. This is because there is evidencethat the performance is better running in serial than in parallel. See http://www.ehu.eus/ehusfera/hpc/2012/06/20/benchmarking-genetic-assemblers-abyss-vs-velvet/ for details.

Installation notes

Velvet was compiled with gcc 4.4.7

tar -xvzf ./velvet_1.2.10.tgzcd velvet_1.2.10make

mkdir -p /usr/local/packages6/apps/gcc/4.4.7/velvet/1.2.10mv * /usr/local/packages6/apps/gcc/4.4.7/velvet/1.2.10

Modulefile

The module file is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/velvet/1.2.10

Its contents are

#%Module1.0######################################################################### velvet 1.2.10 module file##


proc ModulesHelp { } {puts stderr "Makes velvet 1.2.10 available"

}

set VELVET_DIR /usr/local/packages6/apps/gcc/4.4.7/velvet/1.2.10

module-whatis "Makes velevt 1.2.10 available"

prepend-path PATH $VELVET_DIR

xz utils

xz utils


URL http://tukaani.org/xz/

XZ Utils is free general-purpose data compression software with a high compression ratio. XZ Utils were written forPOSIX-like systems, but also work on some not-so-POSIX systems. XZ Utils are the successor to LZMA Utils.


http://www.ehu.eus/ehusfera/hpc/2012/06/20/benchmarking-genetic-assemblers-abyss-vs-velvet/

http://www.ehu.eus/ehusfera/hpc/2012/06/20/benchmarking-genetic-assemblers-abyss-vs-velvet/

http://tukaani.org/xz/


The core of the XZ Utils compression code is based on LZMA SDK, but it has been modified quite a lot to be suitablefor XZ Utils. The primary compression algorithm is currently LZMA2, which is used inside the .xz container format.With typical files, XZ Utils create 30 % smaller output than gzip and 15 % smaller output than bzip2.

XZ Utils consist of several components:

• liblzma is a compression library with an API similar to that of zlib.

• xz is a command line tool with syntax similar to that of gzip.

• xzdec is a decompression-only tool smaller than the full-featured xz tool.

• A set of shell scripts (xzgrep, xzdiff, etc.) have been adapted from gzip to ease viewing, grepping, and comparingcompressed files.

• Emulation of command line tools of LZMA Utils eases transition from LZMA Utils to XZ Utils.

While liblzma has a zlib-like API, liblzma doesn’t include any file I/O functions. A separate I/O library is planned,which would abstract handling of .gz, .bz2, and .xz files with an easy to use API.

Usage

There is an old version of xz utils available on the system by default. We can see its version with

xz --version

which gives

xz (XZ Utils) 4.999.9betaliblzma 4.999.9beta

Version 4.999.9beta of xzutils was released in 2009.

To make version 5.2.2 (released in 2015) available, run the following module command

module load apps/gcc/4.4.7/xzutils/5.2.2

Documentation

Standard man pages are available. The documentation version you get depends on wether or not you’ve loaded themodule.

man xz

Installation notes

This section is primarily for administrators of the system. xz utils 5.2.2 was compiled with gcc 4.4.7

tar -xvzf ./xz-5.2.2.tar.gzcd xz-5.2.2mkdir -p /usr/local/packages6/apps/gcc/4.4.7/xzutils/5.2.2./configure --prefix=/usr/local/packages6/apps/gcc/4.4.7/xzutils/5.2.2makemake install

Testing was performed with

2.2. Iceberg 165


make check

Final piece of output was

==================All 9 tests passed==================

Module file

Modulefile is on the system at /usr/local/modulefiles/apps/gcc/4.4.7/xzutils/5.2.2

#%Module1.0######################################################################### xzutils 5.2.2 module file##


proc ModulesHelp { } {puts stderr "Makes xzutils 5.2.2 available"

}

set XZUTILS_DIR /usr/local/packages6/apps/gcc/4.4.7/xzutils/5.2.2

module-whatis "Makes xzutils 5.2.2 available"

prepend-path PATH $XZUTILS_DIR/binprepend-path LD_LIBRARY_PATH $XZUTILS_DIR/libprepend-path CPLUS_INCLUDE_PATH $XZUTILS_DIR/includeprepend-path CPATH $XZUTILS_DIR/includeprepend-path LIBRARY_PATH $XZUTILS_DIR/libprepend-path MANPATH $XZUTILS_DIR/share/man/

Libraries

FLAMEGPU

FLAME GPU

URL http://www.flamegpu.com/

FLAMEGPU is a multi agent simulation framework developed at the University of Sheffield. It uses GPUs to accel-erate the simulation of multiagent systems but abstracts the GPU architecture away from users so that they can writemodels using a high level of abstraction (without having to write GPU code).

Installing FLAMEGPU on Iceberg

To install FLAME GPU you should checkout the latest master branch of the code which has Linux support


http://www.flamegpu.com/

http://www.flamegpu.com


git clone https://github.com/FLAMEGPU/FLAMEGPU.git

To compile the FLAME GPU examples you will need to be on a GPU node. You can start an interactive session using

qsh -l gpu=1 --l gpu_arch=nvidia-k40m -l rmem=13G

You will then need to load the relevant modules

module load libs/cuda/7.5.18module load compilers/gcc/4.9.2

FLAMEGPU allows you to specify a CUDA_PATH environment variable so that you can change the CUDA versioneasily. To set this for the CUDA 7.5 module use

export CUDA_PATH=/usr/local/packages6/libs/binlibs/CUDA/7.5.18/cuda

You can now navigate to the FLAME GPU examples folder and build the examples. e.g.

cd FLAMEGPU/examplesmake

You can now run the console versions of the example models by navigating to FLAMEGPU/bin/x64 and calling theappropriate shell script. e.g.

cd ../..cd bin/x64./Boids_BruteForce_console.sh

FLAME GPU will run the example for one iteration and output the model state to a file 1.xml in the model’s iterationsdirectory.

Visualisation currently does not work with X forwarding as FLAME GPU uses complex rendering techniques whichare not supported. A solution using VirtualGL is in progress.

beagle

beagle


URL https://github.com/beagle-dev/beagle-lib

Location /usr/local/packages6/libs/gcc/4.4.7/beagle/2.1.2/

General purpose library for evaluating the likelihood of sequence evolution on trees.

Usage

To make this library available, run the following module command:

module load libs/gcc/4.4.7/beagle/2.1.2

This populates the environment variables C_INCLUDE_PATH, LD_LIBRARY_PATH and LD_RUN_PATH with therelevant directories.

2.2. Iceberg 167

https://github.com/beagle-dev/beagle-lib


Installation notes

This section is primarily for administrators of the system.

Beagle 2.1.2 was compiled with gcc 4.4.7.

• Install script

• Module file

Berkeley Lab Checkpoint/Restart (BLCR)

BLCR


URL http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/

From the BLCR User’s Guide:

Checkpoint/Restart allows you to save one or more processes to a file and later restart them from that file. Thereare three main uses for this:

1. Scheduling: Checkpointing a program allows a program to be safely stopped at any point in its execution, sothat some other program can run in its place. The original program can then be run again later.

2. Process Migration: If a compute node appears to be likely to crash, or there is some other reason for shuttingit down (routine maintenance, hardware upgrade, etc.), checkpoint/restart allows any processes running on it tobe moved to a different node (or saved until the original node is available again).

3. Failure recovery: A long-running program can be checkpointed periodically, so that if it crashes due tohardware, system software, or some other non-deterministic cause, it can be restarted from a point in itsexecution more recent that starting from the beginning.

Berkeley Lab Checkpoint/Restart (BLCR) provides checkpoint/restart on Linux systems. BLCR can be used eitherwith processes on a single computer, or on parallel jobs (such as MPI applications) which may be running acrossmultiple machines on a cluster of Linux nodes.

Note: Checkpointing parallel jobs requires a library which has integrated BLCR support. At the present time, manyMPI implementations are known to support checkpoint/restart with BLCR. Consult the corresponding BLCR entry forthe current list.

Usage

BCLR was installed before we started using the module system. As such, it is currently always available to workernodes. It should be considered experimental.

The checkpointing is performed at the kernel level, so any batch code should be checkpointable without modification(it may not work with our MPI environment though, although it should cope with SMP codes).

To run a code, use

cr_run ./executable

To checkpoint a process with process id PID


http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/

https://upc-bugs.lbl.gov/blcr/doc/html/BLCR_Users_Guide.html

https://upc-bugs.lbl.gov/blcr/doc/html/FAQ.html#mpi


cr_checkpoint -f checkpoint.file PID

Use the --term flag if you want to checkpoint and kill the process

To restart the process from a checkpoint

cr_restart checkpoint.file

Using with SGE in batch

A checkpoint environment has been setup called blcr. An example of a checkpointing job would look somethinglike

#!/bin/bash#$ -l h_rt=168:00:00#$ -c sx#$ -ckpt blcrcr_run ./executable >> output.file

The -c sx options tells the scheduler to checkpoint if the queue is suspended, or if the execution daemon is killed.You can also specify checkpoints to occur after a given time period.

A checkpoint file will be produced before the job is terminated. This file will be called checkpoint.[jobid].[pid]. This file will contain the complete in-memory state of your program at the time that it terminates, so makesure that you have enough disk space to save the file.

To resume a checkpointed job submit a job file which looks like

#!/bin/bash#$ -l h_rt=168:00:00#$ -c sx#$ -ckpt blcrcr_restart [name of checkpoint file]

Installation notes

Installation notes are not available.

Boost C++ Library

Boost C++ Library


URL www.boost.org

Boost provides free, peer-reviewed and portable C++ source libraries.

2.2. Iceberg 169


Usage

On Iceberg, different versions of Boost were built using different versions of the gcc compiler, where each versionof GCC provides a potentially different version of the C++ standard library. We therefore suggest that you use thematching version of gcc to build your code.

Boost 1.60.0 was built using gcc 4.9.2. To make both the compiler and Boost library available to the system, executethe following module commands while in a qrsh or qsh session

module load libs/gcc/4.9.2/boost/1.60.0

Boost 1.59 was built using gcc 5.2. To make both the compiler and Boost library available to the system, execute thefollowing module commands while in a qrsh or qsh session

module load compilers/gcc/5.2module load libs/gcc/5.2/boost/1.59

Boost version 1.58 was built using gcc 4.8.2. To make both the compiler and Boost library available to the system,execute the following module commands while in a qrsh or qsh session

module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/boost/1.58

Version 1.41 of Boost uses version 4.4.7 of the gcc compiler. Since this is the default version of gcc on the system,you only need to load the module for the library

module load libs/gcc/4.4.7/boost/1.41

Build a simple program using Boost

Many boost libraries are header-only which makes them particularly simple to compile. The following program readsa sequence of integers from standard input, uses Boost.Lambda to multiply each number by three, and writes them tostandard output (taken from http://www.boost.org/doc/libs/1_58_0/more/getting_started/unix-variants.html):

#include <boost/lambda/lambda.hpp>#include <iostream>#include <iterator>#include <algorithm>

int main(){

using namespace boost::lambda;typedef std::istream_iterator<int> in;

std::for_each(in(std::cin), in(), std::cout << (_1 * 3) << " " );

}

Copy this into a file called example1.cpp and compile with:

g++ example1.cpp -o example

Provided you loaded the correct modules given above, the program should compile without error.


http://www.boost.org/doc/libs/1_58_0/more/getting_started/unix-variants.html


Linking to a Boost library

The following program is taken from the official Boost documentation http://www.boost.org/doc/libs/1_58_0/more/getting_started/unix-variants.html

#include <boost/regex.hpp>#include <iostream>#include <string>

int main(){

std::string line;boost::regex pat( "^Subject: (Re: |Aw: )*(.*)" );

while (std::cin){

std::getline(std::cin, line);boost::smatch matches;if (boost::regex_match(line, matches, pat))

std::cout << matches[2] << std::endl;}

}

This program makes use of the Boost.Regex library, which has a separately-compiled binary component we need tolink to. Assuming that the above program is called example2.cpp, compile with the following command:

g++ example2.cpp -o example2 -lboost_regex

If you get an error message that looks like this:

example2.cpp:1:27: error: boost/regex.hpp: No such file or directory

the most likely cause is that you forgot to load the correct modules as detailed above.

Installation Notes

This section is primarily for administrators of the system

version 1.59: Compiled with gcc 5.2 and icu version 55

module load compilers/gcc/5.2module load libs/gcc/4.8.2/libunistring/0.9.5module load libs/gcc/4.8.2/icu/55

mkdir -p /usr/local/packages6/libs/gcc/5.2/boost/1.59.0/tar -xvzf ./boost_1_59_0.tar.gzcd boost_1_59_0./bootstrap.sh --prefix=/usr/local/packages6/libs/gcc/5.2/boost/1.59.0/

It complained that it could not find the icu library but when I ran

./b2 install --prefix=/usr/local/packages6/libs/gcc/5.2/boost/1.59.0/

It said that it had detected the icu library and was compiling it in

Version 1.58: Compiled with gcc 4.8.2 and icu version 55

2.2. Iceberg 171




module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/libunistring/0.9.5module load libs/gcc/4.8.2/icu/55tar -xvzf ./boost_1_58_0.tar.gzcd boost_1_58_0./bootstrap.sh --prefix=/usr/local/packages6/libs/gcc/4.8.2/boost/1.58.0/

It complained that it could not find the icu library but when I ran

./b2 install --prefix=/usr/local/packages6/libs/gcc/4.8.2/boost/1.58.0

It said that it had detected the icu library and was compiling it in

Version 1.41: This build of boost was built with gcc 4.4.7 and ICU version 42

module load libs/gcc/4.4.7/icu/42tar -xvzf ./boost_1_41_0.tar.gzcd boost_1_41_0./bootstrap.sh --prefix=/usr/local/packages6/libs/gcc/4.4.7/boost/1.41./bjam -sICU_PATH=/usr/local/packages6/libs/gcc/4.4.7/icu/42 install

Testing

The two examples above were compiled and run.

Module Files

Version 1.60.0

Module file location: /usr/local/modulefiles/libs/gcc/4.9.2/boost/1.60.0

#%Module1.0######################################################################### boost 1.60.0 module file##


set vers 1.60.0set gccvers 4.9.2

module load compilers/gcc/$gccversmodule load libs/gcc/4.8.2/libunistring/0.9.5module load libs/gcc/$gccvers/icu/58.1

proc ModulesHelp { } {global versglobal gccversputs stderr "Makes the Boost $vers library (plus gcc $gccvers) available"

}module-whatis puts stderr "Makes the Boost $vers library (plus gcc $gccvers) available→˓"



set BOOST_DIR /usr/local/packages6/libs/gcc/$gccvers/boost/$vers

prepend-path LD_LIBRARY_PATH $BOOST_DIR/libprepend-path CPLUS_INCLUDE_PATH $BOOST_DIR/includeprepend-path LIBRARY_PATH $BOOST_DIR/lib

Version 1.59

Module file location: /usr/local/modulefiles/libs/gcc/5.2/boost/1.59

#%Module1.0######################################################################### boost 1.59 module file##


module load libs/gcc/4.8.2/libunistring/0.9.5module load libs/gcc/4.8.2/icu/55

proc ModulesHelp { } {puts stderr "Makes the Boost 1.59 library available"

}

set BOOST_DIR /usr/local/packages6/libs/gcc/5.2/boost/1.59.0

module-whatis "Makes the Boost 1.59 library available"


Version 1.58

Module file location: /usr/local/modulefiles/libs/gcc/4.8.2/boost/1.58

#%Module1.0######################################################################### boost 1.58 module file##


module load libs/gcc/4.8.2/libunistring/0.9.5module load libs/gcc/4.8.2/icu/55


}

set BOOST_DIR /usr/local/packages6/libs/gcc/4.8.2/boost/1.58.0


prepend-path LD_LIBRARY_PATH $BOOST_DIR/lib

2.2. Iceberg 173


prepend-path CPLUS_INCLUDE_PATH $BOOST_DIR/includeprepend-path LIBRARY_PATH $BOOST_DIR/lib

Version 1.41

The module file is on the system at /usr/local/modulefiles/libs/gcc/4.4.7/boost/1.41

#%Module1.0######################################################################### Boost 1.41 module file##


module load libs/gcc/4.4.7/icu/42


}

set BOOST_DIR /usr/local/packages6/libs/gcc/4.4.7/boost/1.41



bzip2

bzip2 is a freely available, patent free (see below), high-quality data compressor. It typically compresses files to within10% to 15% of the best available techniques (the PPM family of statistical compressors), whilst being around twice asfast at compression and six times faster at decompression.

bzip2

Version 1.0.6

URL http://www.bzip.org/

Location /usr/local/packages6/libs/gcc/4.4.7/bzip2/1.0.6

Usage

The default version of bzip2 on the system is version 1.0.5. If you need a newer version run the following modulecommand

module load libs/gcc/4.4.7/bzip2/1.0.6

Check the version number that’s available using


http://www.bzip.org/


bzip2 --version

Documentation

Standard man pages are available

man bzip2

Installation notes


It was built with gcc 4.4.7

wget http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gzmkdir -p /usr/local/packages6/libs/gcc/4.4.7/bzip2/1.0.6tar -xvzf ./bzip2-1.0.6.tar.gzcd bzip2-1.0.6

make -f Makefile-libbz2_somakemake install PREFIX=/usr/local/packages6/libs/gcc/4.4.7/bzip2/1.0.6mv *.so* /usr/local/packages6/libs/gcc/4.4.7/bzip2/1.0.6/lib/

testing

The library is automatically tested when you do a make. The results were

Doing 6 tests (3 compress, 3 uncompress) ...If there's a problem, things might stop at this point.

./bzip2 -1 < sample1.ref > sample1.rb2



./bzip2 -d < sample1.bz2 > sample1.tst

./bzip2 -d < sample2.bz2 > sample2.tst

./bzip2 -ds < sample3.bz2 > sample3.tstcmp sample1.bz2 sample1.rb2cmp sample2.bz2 sample2.rb2cmp sample3.bz2 sample3.rb2cmp sample1.tst sample1.refcmp sample2.tst sample2.refcmp sample3.tst sample3.ref

If you got this far and the 'cmp's didn't complain, it lookslike you're in business.

Module File

Module location is /usr/local/modulefiles/libs/gcc/4.4.7/bzip2/1.0.6. Module contents

2.2. Iceberg 175


#%Module1.0######################################################################### bzip2 1.0.6 module file##


proc ModulesHelp { } {puts stderr "Makes the bzip 1.0.6 library available"

}

module-whatis "Makes the bzip 1.0.6 library available"

set BZIP_DIR /usr/local/packages6/libs/gcc/4.4.7/bzip2/1.0.6

prepend-path LD_LIBRARY_PATH $BZIP_DIR/libprepend-path CPATH $BZIP_DIR/includeprepend-path MANPATH $BZIP_DIR/manprepend-path PATH $BZIP_DIR/bin

cfitsio

cfitsio

Latest version 3.380

URL http://heasarc.gsfc.nasa.gov/fitsio/fitsio.html

CFITSIO is a library of C and Fortran subroutines for reading and writing data files in FITS (Flexible Image TransportSystem) data format. CFITSIO provides simple high-level routines for reading and writing FITS files that insulate theprogrammer from the internal complexities of the FITS format.

Usage

To make this library available, run the following module command

module load libs/gcc/5.2/cfitsio

The modulefile creates a variable $CFITSIO_INCLUDE_PATH which is the path to the include directory.

Installation notes

This section is primarily for administrators of the system. CFITSIO 3.380 was compiled with gcc 5.2. The compilationused this script and it is loaded with this modulefile.

cgns


http://heasarc.gsfc.nasa.gov/fitsio/fitsio.html


cgns

Version 3.2.1


Dependancies libs/hdf5/gcc/openmpi/1.8.14

URL http://cgns.github.io/WhatIsCGNS.html

Location /usr/local/packages6/libs/gcc/4.4.7/cgnslib

The CFD General Notation System (CGNS) provides a general, portable, and extensible standard for the storage andretrieval of computational fluid dynamics (CFD) analysis data.

Usage


module load libs/gcc/4.4.7/cgns/3.2.1

This will also load the module files for the prerequisite libraries, Open MPI 1.8.3 and HDF5 1.8.14 with parallelsupport.

Installing


• This is a prerequisite for Code Saturne version 4.0.

• It was built with gcc 4.4.7, openmpi 1.8.3 and hdf 1.8.14

module load libs/hdf5/gcc/openmpi/1.8.14tar -xvzf cgnslib_3.2.1.tar.gzmkdir /usr/local/packages6/libs/gcc/4.4.7/cgnslibcd /usr/local/packages6/libs/gcc/4.4.7/cgnslibmkdir 3.2.1cd 3.2.1cmake ~/cgnslib_3.2.1/ccmake .

Configured the following using ccmake

CGNS_ENABLE_PARALLEL ONMPIEXEC /usr/local/mpi/gcc/openmpi/1.8.3/bin/mpiexecMPI_COMPILER /usr/local/mpi/gcc/openmpi/1.8.3/bin/mpic++MPI_EXTRA_LIBRARY /usr/local/mpi/gcc/openmpi/1.8.3/lib/libmpi.sMPI_INCLUDE_PATH /usr/local/mpi/gcc/openmpi/1.8.3/includeMPI_LIBRARY /usr/local/mpi/gcc/openmpi/1.8.3/lib/libmpi_cZLIB_LIBRARY /usr/lib64/libz.so

FORTRAN_NAMING LOWERCASE_HDF5_INCLUDE_PATH /usr/local/packages6/hdf5/gcc-4.4.7/openmpi-1.8.3/→˓hdf5-1.8.14/include/HDF5_LIBRARY /usr/local/packages6/hdf5/gcc-4.4.7/openmpi-1.8.3/→˓hdf5-1.8.14/lib/libhdf5.soHDF5_NEED_MPI ON

2.2. Iceberg 177

http://cgns.github.io/WhatIsCGNS.html


HDF5_NEED_SZIP OFFHDF5_NEED_ZLIB ONCGNS_BUILD_CGNSTOOLS OFFCGNS_BUILD_SHARED ONCGNS_ENABLE_64BIT ONCGNS_ENABLE_FORTRAN ONCGNS_ENABLE_HDF5 ONCGNS_ENABLE_SCOPING OFFCGNS_ENABLE_TESTS ONCGNS_USE_SHARED ONCMAKE_BUILD_TYPE ReleaseCMAKE_INSTALL_PREFIX /usr/local/packages6/libs/gcc/4.4.7/cgnslib/3.2.1

Once the configuration was complete, I did

makemake install

Module File

Module File Location: /usr/local/modulefiles/libs/gcc/4.4.7/cgns/3.2.1

#%Module1.0######################################################################### cgns 3.2.1 module file##


proc ModulesHelp { } {puts stderr "Makes the cgns 3.2.1 library available"

}

module-whatis "Makes the cgns 3.2.1 library available"module load libs/hdf5/gcc/openmpi/1.8.14

set CGNS_DIR /usr/local/packages6/libs/gcc/4.4.7/cgnslib/3.2.1

prepend-path LD_LIBRARY_PATH $CGNS_DIR/libprepend-path CPATH $CGNS_DIR/include

CUDA

CUDA, which stands for Compute Unified Device Architecture, is a parallel computing platform and application pro-gramming interface (API) model created by NVIDIA. It allows software developers to use a CUDA-enabled graphicsprocessing unit (GPU) for general purpose processing - an approach known as General Purpose GPU (GPGPU).

Usage

There are several versions of the CUDA library available. As with many libraries installed on the system, CUDAlibraries are made available via module commands which are only available once you have started a qrshx with a



GPU resource -l gpu=. For example, to request a single GPU:

qrshx -l gpu=1

See Using GPUs on Iceberg for more information on how to request a GPU-enabled node for an interactivesession or job submission.

The latest version CUDA is loaded with the command

module load libs/cuda

Alternatively, you can load a specific version with one of the following:

module load libs/cuda/8.0.44module load libs/cuda/7.5.18module load libs/cuda/6.5.14module load libs/cuda/4.0.17module load libs/cuda/3.2.16

To check which version of CUDA you are using

$ nvcc --versionnvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2016 NVIDIA CorporationBuilt on Sun_Sep__4_22:14:01_CDT_2016Cuda compilation tools, release 8.0, V8.0.44

Important To compile CUDA programs you also need a compatible version of the GNU Compiler Collection (gcc).As of version 8.0.44, CUDA is compatible with GCC versions:

• greater than or equal to 4.7.0 (to allow for the use of c++11 features) and

• less than 5.0.0

It is therefore recommended that you load the most recent 4.x version of GCC when building CUDA programs onIceberg:

module load compilers/gcc/4.9.2

Compiling the sample programs

You do not need to be using a GPU-enabled node to compile the sample programs but you do need a GPU to run them.

In a qrsh session

# Load modulesmodule load libs/cuda/8.0.44module load compilers/gcc/4.9.2

# Copy CUDA samples to a local directory# It will create a directory called NVIDIA_CUDA-8.0_Samples/mkdir cuda_samplescd cuda_samplescp -r $CUDA_SDK .

# Compile (this will take a while)cd NVIDIA_CUDA-8.0_Samples/make

2.2. Iceberg 179


A basic test is to run one of the resulting binaries, deviceQuery.

Documentation

• CUDA Toolkit Documentation

• The power of C++11 in CUDA 7

CUDA Training

GPUComputing@sheffield provides a self-paced introduction to CUDA training course.

Determining the NVIDIA Driver version

Run the command:

cat /proc/driver/nvidia/version

Example output is

NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.81 Wed Aug 17 22:24:07 PDT 2016GCC version: gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)

Installation notes


Device driver

The NVIDIA device driver is installed and configured using the /etc/init.d/uos-nvidia service.

This service does the following at boot time:

• Check the device driver version and uninstall it then reinstall the target version if required;

• Load the nvidia kernel module;

• Create several device nodes in /dev/.

The NVIDIA device driver is currently version 384.81. The driver installer provides OpenGL libraries.

CUDA 8.0.44

1. The CUDA toolkit binaries and samples were installed using a binary .run file:

cuda_vers="8.0.44"prefix="/usr/local/packages/libs/CUDA/binlibs${cuda_vers}"mkdir -m 2775 -p $prefixchown ${USER}:app-admins $prefixcd /usr/local/media/nvidia/chmod +x cuda_${cuda_vers}_linux.run./cuda_${cuda_vers}_linux.run --toolkit --toolkitpath=${prefix}/cuda \


http://docs.nvidia.com/cuda/index.html#axzz3uLoSltnh

http://devblogs.nvidia.com/parallelforall/power-cpp11-cuda-7/

http://gpucomputing.shef.ac.uk

http://gpucomputing.shef.ac.uk/education/cuda/


--samples --samplespath=${prefix}/samples \--no-opengl-libs -silent

2. This modulefile was installed as /usr/local/modulefiles/libs/cuda/8.0.44

CUDA 7.5.18

CUDA 7.5.18

1. The CUDA toolkit binaries and samples were installed using a binary .run file as per version 8.0.44.

2. This modulefile was installed as /usr/local/modulefiles/libs/cuda/7.5.18

Previous versions

No install notes are available.

cuDNN

cudNN

Dependencies CUDA, gcc

URL https://developer.nvidia.com/cudnn

Documentation https://developer.nvidia.com/cudnn

The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deepneural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backwardconvolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.

Usage

Only GPU-enabled nodes are able to run the library. See Using GPUs on Iceberg for more information on howto request a GPU-enabled node for an interactive session or job submission.

Currently cuDNN 5.1 is available for CUDA versions 8.0.44 and 7.5.18. An appropriate CUDA module is loadedautomatically so there’s no need for a separate CUDA module load call.

Load the appropriate cuDNN version with one of the following commands:

module load libs/binlibs/cudnn/5.1-cuda-8.0.44module load libs/binlibs/cudnn/5.1-cuda-7.5.18

Installation notes


• The cuDNN library is only available to download through the developer portal.

• Installation

– Install script: install_cudnn5.1_for_cuda7.5_cuda8.0.sh

2.2. Iceberg 181

https://developer.nvidia.com/cudnn




– Installation .tgz files are located in /usr/local/media/protected/cudnn

• Module file:

– cuDNN 5.1 for CUDA 7.5

– cuDNN 5.1 for CUDA 8.0

curl

curl


URL https://curl.haxx.se/

curl is an open source command line tool and library for transferring data with URL syntax, supporting DICT, FILE,FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB,SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTPform based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5,NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more.

Usage

There is a default version of curl available on the system but it is rather old

curl-config --version

gives the result

libcurl 7.19.7

Version 7.19.7 was released in November 2009!

A newer version of the library is available via the module system. To make it available

module load libs/gcc/4.4.7/curl/7.47.1

The curl-config command will now report the newer version

curl-config --version

Should result in

libcurl 7.47.1

Documentation

Standard man pages are available

man curl


https://curl.haxx.se/


Installation notes


Curl 7.47.1 was compiled with gcc 4.47

• Install script: install_curl_7.47.1.sh

• Module file: 7.47.1

fftw

fftw


URL http://www.fftw.org/

FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, ofarbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sinetransforms or DCT/DST).

Usage

To make this library available, run one of the following module commands:

module load libs/gcc/4.9.2/fftw/3.3.5module load libs/gcc/5.2/fftw/3.3.4

If you are using CUDA then you will want to use version 3.3.5 as the build of version 3.3.4 is dependent on GCC 5.2,a version of GCC not supported by CUDA at this time.

Installation notes


version 3.3.5

This was compiled with GCC 4.9.2 (for compatibility with CUDA, which doesn’t support GCC >= 5.0.0). Threading(inc. OpenMP) and shared-library support were enabled at build-time. Vectorisation support was not explicitly enabledto allow the library to be used on nodes that do not support AVX/AVX2.

First, download, configure, build, test and install using this script.

During the testing stage you should see lots of numerical output plus:

--------------------------------------------------------------FFTW transforms passed basic tests!

--------------------------------------------------------------

--------------------------------------------------------------FFTW threaded transforms passed basic tests!

--------------------------------------------------------------

2.2. Iceberg 183

http://www.fftw.org/


Next, this modulefile as /usr/local/modulefiles/libs/gcc/4.9.2/fftw/3.3.5

version 3.3.4

This was compiled with GCC 5.2:

module load compilers/gcc/5.2mkdir -p /usr/local/packages6/libs/gcc/5.2/fftw/3.3.4tar -xvzf fftw-3.3.4.tar.gzcd fftw-3.3.4./configure --prefix=/usr/local/packages6/libs/gcc/5.2/fftw/3.3.4 --enable-threads --→˓enable-openmp --enable-sharedmakemake check

The result was lots of numerical output and:


--------------------------------------------------------------


--------------------------------------------------------------

Installed with:

make install

The modulefile is on the system at /usr/local/modulefiles/libs/gcc/5.2/fftw/3.3.4

#%Module1.0######################################################################### fftw 3.3.4 module file##


proc ModulesHelp { } {puts stderr "Makes the FFTW 3.3.4 library available"

}module-whatis "Makes the FFTW 3.3.4 library available"

set FFTW_DIR /usr/local/packages6/libs/gcc/5.2/fftw/3.3.4

prepend-path CPLUS_INCLUDE_PATH $FFTW_DIR/includeprepend-path LD_LIBRARY_PATH $FFTW_DIR/libprepend-path LIBRARY_PATH $FFTW_DIR/libprepend-path MANPATH $FFTW_DIR/share/manprepend-path PATH $FFTW_DIR/bin

FLTK



FLTK

Version 1.3.3

URL http://www.fltk.org/index.php

FLTK (pronounced “fulltick”) is a cross-platform C++ GUI toolkit for UNIX/Linux (X11), Windows, and OS X.FLTK provides modern GUI functionality without the bloat and supports 3D graphics via OpenGL and its built-inGLUT emulation.

Usage

Run one of the following:

module load libs/gcc/5.2/fltk/1.3.3module load libs/gcc/4.9.2/fltk/1.3.3

If you are using CUDA then you will want to use the version that was built with (and has a runtime dependency on)GCC 4.9.2 as GCC 5.0 and later are not supported by CUDA at this time.

Installation notes


Version 1.3.3 (GCC 4.9.2)

• Compiled with (and has a runtime dependency on) GCC 4.9.2 (for compatibility with CUDA, which doesn’tsupport GCC >= 5.0.0).

• Shared-library support and X Font Library (Xft) support were enabled at build-time.

• A pre-requisite for Relion 2-beta.


Next, this modulefile as /usr/local/modulefiles/libs/gcc/4.9.2/fltk/1.3.3

Version 1.3.3 (GCC 5.2)

• This is a pre-requisite for GNU Octave version 4.0

• Compiled with (and has a runtime dependency on) GCC 5.2

• Installed using:

module load compilers/gcc/5.2mkdir -p /usr/local/packages6/libs/gcc/5.2/fltk/1.3.3#tar -xvzf ./fltk-1.3.3-source.tar.gzcd fltk-1.3.3

#Fixes error relating to undefined _ZN18Fl_XFont_On_Demand5valueEv#Source https://groups.google.com/forum/#!topic/fltkgeneral/GT6i2KGCb3Ased -i 's/class Fl_XFont_On_Demand/class FL_EXPORT Fl_XFont_On_Demand/' FL/x.H

./configure --prefix=/usr/local/packages6/libs/gcc/5.2/fltk/1.3.3 --enable-shared→˓--enable-xftmakemake install

2.2. Iceberg 185

http://www.fltk.org/index.php

https://www.freedesktop.org/wiki/Software/Xft/


Modulefile at /usr/local/modulefiles/libs/gcc/5.2/fltk/1.3.3:

#%Module1.0######################################################################### fltk 1.3.3 module file##



proc ModulesHelp { } {puts stderr "Makes the FLTK 1.3.3 library available"

}

set FLTK_DIR /usr/local/packages6/libs/gcc/5.2/fltk/1.3.3

module-whatis "Makes the FLTK 1.3.3 library available"

prepend-path LD_LIBRARY_PATH $FLTK_DIR/libprepend-path CPLUS_INCLUDE_PATH $FLTK_DIR/includeprepend-path LIBRARY_PATH $FLTK_DIR/libprepend-path PATH $FLTK_DIR/bin

Geospatial Data Abstraction Library (GDAL)

GDAL


URL http://www.gdal.org/

GDAL is a library used by many Geographic Information Systems (GIS) packages for converting between manydifferent raster and vector GIS data formats. It also includes command-line utilities for data translation and processing.It is released under an an X/MIT style Open Source license by the Open Source Geospatial Foundation.

Usage

By running

$ module load libs/gcc/6.2/proj/2.1.1

you

• add several GDAL programs to your PATH environment variable

• allow other programs to make use of (dynamically link against) the GDAL library

You can run gdal-config --version to test that you are running the required version

$ gdal-config --version2.1.1


http://www.gdal.org/

http://www.osgeo.org


The command-line programs that GDAL provides are

* ``gdaladdo``

* `gdalbuildvrt``

* `gdal-config``

* `gdal_contour``

* `gdaldem``

* `gdalenhance``

* `gdal_grid``

* `gdalinfo``

* `gdallocationinfo``

* `gdalmanage``

* `gdal_rasterize``

* `gdalserver``

* `gdalsrsinfo``

* `gdaltindex``

* `gdaltransform``

* `gdal_translate``

* `gdalwarp``

* `nearblack``

* ògr2ogr``

* ògrinfo``

* ògrlineref``

* ògrtindex``

* `testepsg``

The gdal... utilities are mostly for processing raster GIS data formats, whilst the ogr... utilities are for process-ing vector GIS data formats.

Documentation

Standard man pages are available for the provided commands/functions.

These can be viewed using e.g.

$ man gdal_contour

Much more information is available on the project site.

Supported file formats

GDAL has been compiled on this system with support for only a limited set of GIS data formats. See Installationnotes below for a list of those provided by each available version of GDAL.

Installation notes


Version 2.1.1

GDAL 2.1.1 was compiled with v6.2.0 of the GCC compiler suite.

1. cd to a scratch directory.

2. Download, build and install GDAL using install.sh, ensuring that all output is redirected into a log file.GDAL files are installed into /usr/local/packages6/libs/gcc/6.2/gdal/2.1.1/

2.2. Iceberg 187



3. Install this modulefile as /usr/local/modulefiles/libs/gcc/6.2/gdal/2.1.1

The file formats supported by the build generated using install.sh are listed here.

geos

geos

Version 3.4.2


Dependancies compilers/gcc/4.8.2

URL http://trac.osgeo.org/geos/

Location /usr/local/packages6/libs/gcc/4.8.2/geos/3.4.2

GEOS (Geometry Engine, Open Source) is a C/C++ port of a subset of the Java Topology Suite (JTS), which in turnis a library that:

> provides an object model for planar geometry together with a set of fundamental geometric functions. > JTSconforms to the Simple Features Specification for SQL published by the Open GIS Consortium. > JTS is designed tobe used as a core component of vector-based geomatics software such as geographical information systems. > It canalso be used as a general-purpose library providing algorithms in computational geometry.

Usage

To make this library available, run the following module commands

module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/geos/3.4.2

We load version 4.8.2 of gcc since gcc 4.8.2 was used to build this version of geos.

The rgeos interface in R

rgeos is a CRAN package that provides an R interface to geos. It is not installed in R by default so you need to installa version in your home directory.

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qrsh or qshcommand. Run the following module commands

module load apps/R/3.2.0module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/geos/3.4.2

Launch R and run the command

install.packages('rgeos')

If you’ve never installed an R package before on the system, it will ask you if you want to install to a personal library.Answer y to any questions you are asked.


http://trac.osgeo.org/geos/

http://locationtech.github.io/jts/

http://www.opengeospatial.org/standards/sfs


The library will be installed to a sub-directory called R in your home directory and you should only need to performthe above procedure once.

Once you have performed the installation, you will only need to run the module commands above to make the geoslibrary available to the system. Then, you use rgeos as you would any other library in R

library('rgeos')

Installation notes


qrshtar -xvjf ./geos-3.4.2.tar.bz2cd geos-3.4.2mkdir -p /usr/local/packages6/libs/gcc/4.8.2/geos/3.4.2module load compilers/gcc/4.8.2./configure prefix=/usr/local/packages6/libs/gcc/4.8.2/geos/3.4.2

Potentially useful output at the end of the configure run

Swig: falsePython bindings: falseRuby bindings: falsePHP bindings: false


makemake install

Testing

Compile and run the test-suite with

make check

All tests passed.

Module File

Module File Location: /usr/local/modulefiles/libs/gcc/4.8.2/geos/3.4.2

more /usr/local/modulefiles/libs/gcc/4.8.2/geos/3.4.2#%Module1.0######################################################################### geos 3.4.2 module file##


2.2. Iceberg 189


proc ModulesHelp { } {puts stderr "Makes the geos 3.4.2 library available"

}

set GEOS_DIR /usr/local/packages6/libs/gcc/4.8.2/geos/3.4.2

module-whatis "Makes the geos 3.4.2 library available"

prepend-path LD_LIBRARY_PATH $GEOS_DIR/libprepend-path PATH $GEOS_DIR/bin

GNU Scientific Libary (GSL)

GSL

Latest version 2.3

URL https://www.gnu.org/software/gsl

The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers. It is free software under theGNU General Public License.

The library provides a wide range of mathematical routines such as random number generators, special functions andleast-squares fitting. There are over 1000 functions in total with an extensive test suite.

Usage

You can make the GSL available by running:

$ module load libs/gcc/4.9.2/gsl/2.3

This makes the GSL’s library files, header files, utility programs and documentation all accessible.

YOu can check that you are running the requested version:

$ gsl-config --version2.3

When building software that uses the GSL you will need to link to it. From the GSL documentation:

To link against the library you need to specify both the main library and a supporting CBLAS library,which provides standard basic linear algebra subroutines. A suitable CBLAS implementation is providedin the library libgslcblas.a if your system does not provide one. The following example shows howto link an application with the library:

$ $CC example.o -lgsl -lgslcblas -lm

Another CBLAS implementation that you may want to use with the GSL is the Intel Math Kernel Library (MKL).

Documentation

For documentation either run:


https://www.gnu.org/software/gsl

https://www.gnu.org/software/gsl/manual/html_node/Linking-programs-with-the-library.html#Linking-programs-with-the-library


$ module load libs/gcc/4.9.2/gsl/2.3$ info gsl

or visit the project website.

Installation notes


The GSL was installed as a user wanted to build EIGENSOFT.

Version 2.3

This was compiled using GCC on an Intel x5650 node.


2. Run this install script (install.sh)

3. Install this module file as /usr/local/modulefiles/libs/gcc/4.9.2/gsl/2.3

HDF5

HDF5

Version 1.8.16, 1.8.15-patch1, 1.8.14 and 1.8.13

Dependencies gcc or pgi compiler, openmpi (optional)

URL http://www.hdfgroup.org/HDF5/

Documentation http://www.hdfgroup.org/HDF5/doc/

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety ofdatatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable andis extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools andapplications for managing, manipulating, viewing, and analyzing data in the HDF5 format.

Two primary versions of this library are provided, MPI parallel enabled versions and serial versions.

Usage - Serial

The serial versions were built with gcc version 4.8.2. As such, if you are going to build anything against these versionsof HDF5, we recommend that you use gcc 4.8.2 which can be enabled with the following module command


To enable the serial version of HDF5, use one of the following module commands depending on which version of thelibrary you require:

module load libs/hdf5/gcc/1.8.14module load libs/hdf5/gcc/1.8.13

2.2. Iceberg 191

https://github.com/DReichLab/EIG

http://www.hdfgroup.org/HDF5/

http://www.hdfgroup.org/HDF5/doc/


Usage – Parallel

There are multiple versions of parallel HDF5 installed with different openmpi and compiler versions.

Two versions of HDF were built using gcc version 4.4.7 and OpenMPI version 1.8.3. Version 4.4.7 of gcc is the defaultcompiler on the system so no module command is required for this

module load libs/hdf5/gcc/openmpi/1.8.14module load libs/hdf5/gcc/openmpi/1.8.13

One version was built with the PGI compiler version 15.7 and openmpi version 1.8.8

module load libs/hdf5/pgi/1.8.15-patch1

The above module also loads the relevant modules for OpenMPI and PGI Compiler. To see which modules have beenloaded, use the command module list

Finally, another version was built with GCC 4.4.7 and openmpi 1.10.1, this version is also linked against ZLIB andSZIP.:

module load libs/gcc/4.4.7/openmpi/1.10.1/hdf5/1.8.16

Installation notes


Version 1.8.16 built using GCC Compiler, with seperate ZLIB and SZIP

• install_hdf5.sh Install script

• 1.8.16

Version 1.8.15-patch1 built using PGI Compiler

Here are the build details for the module libs/hdf5/pgi/1.8.15-patch1

Compiled using PGI 15.7 and OpenMPI 1.8.8

• install_pgi_hdf5_1.8.15-patch1.sh Install script

• Modulefile located on the system at /usr/local/modulefiles/libs/hdf5/pgi/1.8.15-patch1

gcc versions

This package is built from the source code distribution from the HDF Group website.

Two primary versions of this library are provided, a MPI parallel enabled version and a serial version. The serialversion has the following configuration flags enabled:

--enable-fortran --enable-fortran2003 --enable-cxx --enable-shared

The parallel version has the following flags:

--enable-fortran --enable-fortran2003 --enable-shared --enable-parallel

The parallel library does not support C++, hence it being disabled for the parallel build.



icu - International Components for Unicode

icu

Latest Version 58.1

URL http://site.icu-project.org/

ICU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for softwareapplications. ICU is widely portable and gives applications the same results on all platforms and between C/C++ andJava software.

ICU is released under a nonrestrictive open source license that is suitable for use with both commercial software andwith other open source or free software.

Usage

Version 58.1 of the icu library for C requires gcc version 4.9.2 (for the C++ standard library); To make the library andthis compiler available, run the following:

module load libs/gcc/4.9.2/icu/58.1

Version 55 of the icu library for C requires gcc version 4.8.2. To make the compiler and library available, run thefollowing module commands:

module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/icu/55

Version 42 of the icu library for C uses gcc version 4.4.7 which is the default on Iceberg. To make the library available,run the following command:

module load libs/gcc/4.4.7/icu/42

Installation Notes


Version 58.1

This Icu 58.1 build links against the GCC 4.9.2 C++ standard library and was installed as a dependency ofboost_iceberg (build using the same C++ standard library); Boost in turn was installed as a dependency of Gromacs2016.1.

module load compilers/gcc/4.9.2tar -xvzf icu4c-58_1-src.tgzcd icu/source./runConfigureICU Linux/gcc --prefix=/usr/local/packages6/libs/gcc/4.9.2/icu/58.1/makemake checkmake install

2.2. Iceberg 193

http://site.icu-project.org/


Version 55

Icu 55 is a pre-requisite for the version of boost_iceberg required for an experimental R module used by one of ourusers. It was built using gcc 4.8.2.

module load compilers/gcc/4.8.2tar -xvzf icu4c-55_1-src.tgzcd icu/source./runConfigureICU Linux/gcc --prefix=/usr/local/packages6/libs/gcc/4.8.2/icu/55/makemake checkmake install

Version 42

Icu version 42 was originally installed as a system RPM. This install moved icu to a module-based install.

tar -xvzf icu4c-4_2_1-src.tgzcd icu/source/./runConfigureICU Linux/gcc --prefix=/usr/local/packages6/libs/gcc/4.4.7/icu/42makemake checkmake install

Testing

Version 58.1

Last few lines of output from make check were:

ALL TESTS SUMMARY:All tests OK: testdata intltest iotest cintltstmake[1]: Leaving directory `/home/te1st/icu_gcc_4_9_2/icu/source/test'make[1]: Entering directory `/home/te1st/icu_gcc_4_9_2/icu/source'verifying that icu-config --selfcheck can operateverifying that make -f Makefile.inc selfcheck can operatePASS: config selfcheck OK

Version 55


[All tests passed successfully...]Elapsed Time: 00:00:00.086make[2]: Leaving directory `/home/te1st/icu/icu/source/test/letest'---------------ALL TESTS SUMMARY:All tests OK: testdata intltest iotest cintltst letestmake[1]: Leaving directory `/home/te1st/icu/icu/source/test'make[1]: Entering directory `/home/te1st/icu/icu/source'verifying that icu-config --selfcheck can operateverifying that make -f Makefile.inc selfcheck can operate



PASS: config selfcheck OKrm -rf test-local.xml

Version 42


All tests passed successfully...]Elapsed Time: 00:00:12.000make[2]: Leaving directory `/home/te1st/icu/source/test/cintltst'---------------ALL TESTS SUMMARY:ok: testdata iotest cintltst===== ERRS: intltestmake[1]: *** [check-recursive] Error 1make[1]: Leaving directory `/home/te1st/icu/source/test'make: *** [check-recursive] Error 2

The error can be ignored since it is a bug in the test itself.

Module Files

Version 58.1

Module File Location: /usr/local/modulefiles/libs/gcc/4.9.2/icu/58.1

#%Module1.0######################################################################### icu 58.1 module file##


set vers 58.1set gccvers 4.9.2

proc ModulesHelp { } {global versglobal gccversputs stderr "Makes icu library $vers (and GCC $gccvers) available"

}module-whatis "Makes icu library $vers (and GCC $gccvers) available"

# Run-time dependency on C++ std libmodule load compilers/gcc/$gccvers

set ICU_DIR /usr/local/packages6/libs/gcc/$gccvers/icu/$vers

prepend-path LD_LIBRARY_PATH $ICU_DIR/libprepend-path LIBRARY_PATH $ICU_DIR/libprepend-path CPATH $ICU_DIR/include

2.2. Iceberg 195

http://sourceforge.net/p/icu/mailman/message/32443311/


Version 55

Module File Location: /usr/local/modulefiles/libs/gcc/4.8.2/icu/55

#%Module1.0######################################################################### icu 55 module file##


proc ModulesHelp { } {puts stderr "Makes the icu library available"

}

set ICU_DIR /usr/local/packages6/libs/gcc/4.8.2/icu/55

module-whatis "Makes the icu library available"

prepend-path LD_LIBRARY_PATH $ICU_DIR/libprepend-path LIBRARY_PATH $ICU_DIR/libprepend-path CPLUS_INCLUDE_PATH $ICU_DIR/include

Version 42

Module File Location: /usr/local/modulefiles/libs/gcc/4.4.7/icu/42

#%Module1.0######################################################################### icu 42 module file##


proc ModulesHelp { } {puts stderr "Makes the icu library available"

}

set ICU_DIR /usr/local/packages6/libs/gcc/4.4.7/icu/42

module-whatis "Makes the icu library available"

prepend-path LD_LIBRARY_PATH $ICU_DIR/libprepend-path LIBRARY_PATH $ICU_DIR/libprepend-path CPLUS_INCLUDE_PATH $ICU_DIR/include

Intel Data Analytics Acceleration Library

Intel’s Data Analytics Acceleration Library (DAAL) provides functions for data analysis (characterization, summa-rization, and transformation) and machine learning (regression, classification, and more).



Parallel Studio Composer Edition version

DAAL can be used with and without other Parallel Studio packages. To access it:

module load libs/binlibs/intel-daal/2017.0

Licensing and availability

See the information on Parallel Studio licensing.

Installation Notes

The following notes are primarily for system administrators.

Intel DAAL 2017.0

Installed as part of Parallel Studio Composer Edition 2017.

This modulefile was installed as /usr/local/modulefiles/libs/binlibs/intel-daal/2017.0.

Intel Integrated Performance Primitives

Integrated Performance Primitives (IPP) are “high-quality, production-ready, low-level building blocks for image pro-cessing, signal processing, and data processing (data compression/decompression and cryptography) applications.”


IPP can be used with and without other Parallel Studio packages. To access it:

module load libs/binlibs/intel-ipp/2017.0



Installation Notes


Intel IPP 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/binlibs/intel-ipp/2017.0/binary.

2.2. Iceberg 197


Intel Math Kernel Library

Intel’s Math Kernel Library (MKL) provides highly optimized, threaded and vectorized functions to maximize per-formance on each processor family. It Utilises de-facto standard C and Fortran APIs for compatibility with BLAS,LAPACK and FFTW functions from other math libraries.

Interactive usage

The MKL was installed along with the Intel compilers as part of Intel Parallel Studio. The MKL can be used with orwithout other Parallel Studio packages (such as the Intel compilers).

To activate just the MKL, use one of :

module load libs/binlibs/intel-mkl/2017.0

module load libs/binlibs/intel-mkl/11.2.3

Note that 2017.0 is newer than 11.2.3.

Sample C and Fortran programs demonstrating matrix multiplication are available for version 2017.0 in the directory$MKL_SAMPLES:

$ ls $MKL_SAMPLES/mkl_c_samples mkl_fortran_samples

To compile one of these you need to copy the samples to a directory you can write to, ensure the MKL and the Intelcompilers are both loaded, change into the directory containing the relevant sample program (C or Fortran) then runmake to compile:

$ qrsh$ cp -r $MKL_SAMPLES/ ~/mkl_samples$ module load libs/binlibs/intel-mkl/2017.0$ module load libs/binlibs/intel-mkl/2017.0$ cd ~/mkl_samples/mkl_fortran_samples/matrix_multiplication$ make

ifort -c src/dgemm_example.f -o release/dgemm_example.oifort release/dgemm_example.o -mkl -static-intel -o release/dgemm_exampleifort -c src/dgemm_with_timing.f -o release/dgemm_with_timing.oifort release/dgemm_with_timing.o -mkl -static-intel -o release/dgemm_with_timingifort -c src/matrix_multiplication.f -o release/matrix_multiplication.oifort release/matrix_multiplication.o -mkl -static-intel -o release/matrix_→˓multiplicationifort -c src/dgemm_threading_effect_example.f -o release/dgemm_threading_effect_→˓example.oifort release/dgemm_threading_effect_example.o -mkl -static-intel -o release/dgemm_→˓threading_effect_example

You should then find several compiled programs in the release directory:

$ ./release/matrix_multiplicationThis example measures performance of computing the realmatrix product C=alpha*A*B+beta*C usinga triple nested loop, where A, B, and C are matricesand alpha and beta are double precision scalars

Initializing data for matrix multiplication C=A*B for



matrix A( 2000 x 200) and matrix B( 200 x 1000)

Intializing matrix data

Making the first run of matrix product usingtriple nested loop to get stable run timemeasurements

Measuring performance of matrix product usingtriple nested loop

== Matrix multiplication using triple nested loop ==== completed at 184.42246 milliseconds ==

Example completed.

For documentation and a tutorial, start an interactive session using qsh, load MKL, then run:

$ firefox $MKL_SAMPLES/mkl_fortran_samples/matrix_multiplication/tutorial/en/index.htm

Installation Notes


Intel MKL 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/binlibs/intel-mkl/2017.0.

Intel MKL 11.2.3

Installed as part of Intel Parallel Studio Composer Edition 2015 Update 3.

This modulefile was installed as /usr/local/modulefiles/libs/binlibs/intel-mkl/11.2.3

Intel Threading Building Blocks

Threading Building Blocks (TBB) lets you write “parallel C++ programs that take full advantage of multicore perfor-mance, that are portable and composable, and that have future-proof scalability.”

Interactive usage

TBB can be used with and without other Parallel Studio packages.

To make use of it you first need to start an interactive session that has access to multiple cores. Here we request sixcores from the clusters’ scheduler:

$ qrsh -pe openmp 6

Next, we need to active a specific version of TBB:

$ module load libs/binlibs/intel-tbb/2017.0

2.2. Iceberg 199


You can find sample TBB programs in the directory $TBB_SAMPLES:

$ ls $TBB_SAMPLES

common concurrent_hash_map concurrent_priority_queue GettingStarted graph index.→˓html license.txt parallel_do parallel_for parallel_reduce pipeline task task_→˓arena task_group test_all

We can compile and run a sample program by copying the samples to our (writable) home directory, moving in to thedirectory containing the sample program (which should have a Makefile) then running make:

$ cp -r $TBB_SAMPLES ~/tbb_samples$ cd ~/tbb_samples/GettingStarted/sub_string_finder/$ ls

license.txt Makefile readme.html sub_string_finder.cpp sub_string_finder_extended.→˓cpp sub_string_finder_pretty.cpp

$ make

g++ -O2 -DNDEBUG -o sub_string_finder sub_string_finder.cpp -ltbb -lrtg++ -O2 -DNDEBUG -o sub_string_finder_pretty sub_string_finder_pretty.cpp -ltbb -lrtg++ -O2 -DNDEBUG -o sub_string_finder_extended sub_string_finder_extended.cpp -ltbb -→˓lrt./sub_string_finder_extendedDone building string.Done with serial version.Done with parallel version.Done validating results.

Serial version ran in 8.51745 secondsParallel version ran in 1.51716 seconds

Many of the sample directories contain HTML documentation. To read this you need to start an interactive graphicalsession (using qsh or qrshx) then run:

$ firefox ~/tbb_samples/index.html

Tutorials

See Chrys Woods’ excellent tutorials for C++ programmers and Python programmers.



Installation Notes


Intel TBB 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/binlibs/intel-tbb/2017.0.


http://chryswoods.com/parallel_c++

http://chryswoods.com/parallel_python/index.html


libunistring

libunistring


URL http://www.gnu.org/software/libunistring/

Text files are nowadays usually encoded in Unicode, and may consist of very different scripts - from Latin letters toChinese Hanzi, with many kinds of special characters: accents, right-to-left writing marks, hyphens, Roman numbers,and much more. But the POSIX platform APIs for text do not contain adequate functions for dealing with particularproperties of many Unicode characters. In fact, the POSIX APIs for text have several assumptions at their base whichdon’t hold for Unicode text.

This library provides functions for manipulating Unicode strings and for manipulating C strings according to theUnicode standard.

Usage

To make this library available, run the following module command:

module load libs/gcc/4.8.2/libunistring/0.9.5

This correctly populates the environment variables LD_LIBRARY_PATH, LIBRARY_PATH andCPLUS_INCLUDE_PATH.

Installation notes

This section is primarily for administrators of the system. libunistring 0.9.5 was compiled with gcc 4.8.2

module load compilers/gcc/4.8.2tar -xvzf ./libunistring-0.9.5.tar.gzcd ./libunistring-0.9.5./configure --prefix=/usr/local/packages6/libs/gcc/4.8.2/libunistring/0.9.5

#buildmake#Testingmake check#Installmake install

Testing

Run make check after make and before make install. This runs the test suite.

Results were

============================================================================Testsuite summary for============================================================================# TOTAL: 492

2.2. Iceberg 201

http://www.gnu.org/software/libunistring/


# PASS: 482# SKIP: 10# XFAIL: 0# FAIL: 0# XPASS: 0# ERROR: 0============================================================================

Module file

The module file is on the system at /usr/local/modulefiles/libs/gcc/4.8.2/libunistring/0.9.5

#%Module1.0######################################################################### libunistring 0.9.5 module file##


proc ModulesHelp { } {puts stderr "Makes the libunistring library available"

}

set LIBUNISTRING_DIR /usr/local/packages6/libs/gcc/4.8.2/libunistring/0.9.5

module-whatis "Makes the libunistring library available"

prepend-path LD_LIBRARY_PATH $LIBUNISTRING_DIR/lib/prepend-path LIBRARY_PATH $LIBUNISTRING_DIR/libprepend-path CPLUS_INCLUDE_PATH $LIBUNISTRING_DIR/include

MED

MED

Version 3.0.8


URL http://www.salome-platform.org/downloads/current-version

Location /usr/local/packages6/libs/gcc/4.4.7/med/3.0.8

The purpose of the MED module is to provide a standard for storing and recovering computer data associated tonumerical meshes and fields, and to facilitate the exchange between codes and solvers.


http://www.salome-platform.org/downloads/current-version


Usage


module load libs/gcc/4.4.7/med/3.0.8

Installation notes


• This is a pre-requisite for Code Saturne version 4.0.

• It was built with gcc 4.4.7, openmpi 1.8.3 and hdf5 1.8.14

module load mpi/gcc/openmpi/1.8.3tar -xvzf med-3.0.8.tar.gzcd med-3.0.8mkdir -p /usr/local/packages6/libs/gcc/4.4.7/med/3.0.8./configure --prefix=/usr/local/packages6/libs/gcc/4.4.7/med/3.0.8 --disable-fortran -→˓-with-hdf5=/usr/local/packages6/hdf5/gcc-4.4.7/openmpi-1.8.3/hdf5-1.8.14/ --disable-→˓pythonmakemake install

Fortran was disabled because otherwise the build failed with compilation errors. It’s not needed for Code Saturne 4.0.

Python was disabled because it didn’t have MPI support.

testing

The following was submiited as an SGE job from the med-3.0.8 build directory

#!/bin/bash

#$ -pe openmpi-ib 8#$ -l rmem=6G

module load mpi/gcc/openmpi/1.8.3make check

All tests passed

Module File

#%Module1.0######################################################################### MED 3.0.8 module file##


proc ModulesHelp { } {

2.2. Iceberg 203


puts stderr "Makes the MED 3.0.8 library available"}

module-whatis "Makes the MED 3.0.8 library available"

set MED_DIR /usr/local/packages6/libs/gcc/4.4.7/med/3.0.8

prepend-path LD_LIBRARY_PATH $MED_DIR/lib64prepend-path CPATH $MED_DIR/include

NAG Fortran Library (Serial)

Produced by experts for use in a variety of applications, the NAG Fortran Library has a global reputation for itsexcellence and, with hundreds of fully documented and tested routines, is the largest collection of mathematical andstatistical algorithms available.

This is the serial (1 CPU core) version of the NAG Fortran Library. For many routines, you may find it beneficial touse the parallel version of the library.

Usage

There are several versions of the NAG Fortran Library available. The one you choose depends on which compiler youare using. As with many libraries installed on the system, NAG libraries are made available via module commandswhich are only available once you have started a qrsh or qsh session.

In addition to loading a module for the library, you will usually need to load a module for the compiler you are using.

NAG for Intel Fortran

Use the following command to make Mark 25 of the serial (1 CPU core) version of the NAG Fortran Library for Intelcompilers available

module load libs/intel/15/NAG/fll6i25dcl

Once you have ensured that you have loaded the module for the Intel compilers you can compile your NAG programusing

ifort your_code.f90 -lnag_mkl -o your_code.exe

which links to a version of the NAG library that’s linked against the high performance Intel MKL (which provideshigh-performance versions of the BLAS and LAPACK libraries). Alternatively, you can compile using

ifort your_code.f90 -lnag_nag -o your_code.exe

Which is linked against a reference version of BLAS and LAPACK. If you are in any doubt as to which to choose, wesuggest that you use -lnag_mkl

NAG for PGI Fortran

Use the following command to make Mark 24 of the serial (1 CPU core) version of the NAG Fortran Library for PGIcompilers available

module load libs/pgi/15/NAG/fll6a24dpl

Once you have ensured that you have loaded the module for the PGI Compilers you can compile your NAG programusing



pgf90 your_code.f90 -lnag_mkl -I$NAGINC -o your_code.exe

which links to a version of the NAG library that’s linked against the high performance Intel MKL (which provideshigh-performance versions of the BLAS and LAPACK libraries). Alternatively, you can compile using

pgf90 your_code.f90 -lnag_nag -I$NAGINC -o your_code.exe

Which is linked against a reference version of BLAS and LAPACK. If you are in any doubt as to which to choose, wesuggest that you use -lnag_mkl

Running NAG’s example programs

Most of NAG’s routines come with example programs that show how to use them. When you use the module com-mand to load a version of the NAG library, the script nag_example for that version becomes available. Providingthis script with the name of the NAG routine you are interested in will copy, compile and run the example program forthat routine into your current working directory.

For example, here is an example output for the NAG routine a00aaf which identifies the version of the NAG libraryyou are using. If you try this yourself, the output you get will vary according to which version of the NAG library youare using

nag_example a00aaf

If you have loaded the module for fll6i25dcl this will give the following output

Copying a00aafe.f90 to current directorycp /usr/local/packages6/libs/intel/15/NAG/fll6i25dcl/examples/source/a00aafe.f90 .

Compiling and linking a00aafe.f90 to produce executable a00aafe.exeifort -I/usr/local/packages6/libs/intel/15/NAG/fll6i25dcl/nag_interface_blocks→˓a00aafe.f90 /usr/local/packages6/libs/intel/15/NAG/fll6i25dcl/lib/libnag_nag.a -o→˓a00aafe.exe

Running a00aafe.exe./a00aafe.exe > a00aafe.rA00AAF Example Program Results

*** Start of NAG Library implementation details ***

Implementation title: Linux 64 (Intel 64 / AMD64), Intel Fortran, Double Precision→˓(32-bit integers)

Precision: FORTRAN double precisionProduct Code: FLL6I25DCL

Mark: 25.1.20150610 (self-contained)

*** End of NAG Library implementation details ***

Functionality

The key numerical and statistical capabilities of the Fortran Library are shown below.

• Click here for a complete list of the contents of the Library.

• Cick here to see what’s new in Mark 25 of the library.

Numerical Facilities

2.2. Iceberg 205

http://www.nag.co.uk/numeric/fl/nagdoc_fl25/html/FRONTMATTER/manconts.html

http://www.nag.co.uk/numeric/fl/new-functionality


• Optimization, both Local and Global

• Linear, quadratic, integer and nonlinear programming and least squares problems

• Ordinary and partial differential equations, and mesh generation

• Solution of dense, banded and sparse linear equations and eigenvalue problems

• Solution of linear and nonlinear least squares problems

• Curve and surface fitting and interpolation

• Special functions

• Numerical integration and integral equations

• Roots of nonlinear equations (including polynomials)

• Option Pricing Formulae

• Wavelet Transforms

Statistical Facilities

• Random number generation

• Simple calculations on statistical data

• Correlation and regression analysis

• Multivariate methods

• Analysis of variance and contingency table analysis

• Time series analysis

• Nonparametric statistics

Documentation

• The NAG Fortran MK25 Library Manual (Link to NAG’s webbsite)

• The NAG Fortran MK24 Library Manual ( Link to NAG’s website)

Installation notes

fll6i25dcl

These are primarily for system administrators

tar -xvzf ./fll6i25dcl.tgz./install.sh

The installer is interactive. Answer the installer questions as follows

Do you wish to install NAG Mark 25 Library? (yes/no):yes

License file gets shown


http://www.nag.co.uk/numeric/fl/fldocumentation.asp

http://www.nag.co.uk/numeric/fl/nagdoc_fl24/html/frontmatter/manconts.html


[accept/decline]? :accept

Where do you want to install the NAG Fortran Library Mark 25?Press return for default location (/opt/NAG)or enter an alternative path.The directory will be created if it does not already exist.>/usr/local/packages6/libs/intel/15/NAG/

Module Files

fll6i25dcl

• The module file is on the system at /usr/local/modulefiles/libs/intel/15/NAG/fll6i25dcl


fll6a24dpl

• The module file is on the system at /usr/local/modulefiles/libs/pgi/15/NAG/fll6a24dpl

NetCDF (fortran)

NetCDF


Dependancies mpi/gcc/openmpi/1.10.1 libs/gcc/4.4.7/openmpi/1.10.1/hdf5/1.8.16

URL http://www.unidata.ucar.edu/software/netcdf/

Documentation http://www.unidata.ucar.edu/software/netcdf/docs/

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation,access, and sharing of array-oriented scientific data. NetCDF was developed and is maintained at Unidata. Unidataprovides data and software tools for use in geoscience education and research. Unidata is part of the UniversityCorporation for Atmospheric Research (UCAR) Community Programs (UCP). Unidata is funded primarily by theNational Science Foundation.

Usage

This version of NetCDF was compiled using version 4.4.7 of the gcc compiler, openmpi 1.10.1 and HDF5 1.8.16. Tomake it available, run the following module command after starting a qsh or qrsh session

module load libs/gcc/4.4.7/openmpi/1.10.1/netcdf-fortran/4.4.3

This module also loads the relevant modules for OpenMPI, PGI Compiler and HDF5. To see which modules havebeen loaded, use the command module list

2.2. Iceberg 207

http://www.unidata.ucar.edu/software/netcdf/

http://www.unidata.ucar.edu/software/netcdf/docs/


Installation notes


Version 4.4.3

• install_gcc_netcdf-fortran_4.4.3.sh Install script

• Modulefile.

NetCDF (PGI build)

NetCDF

Latest version

Dependancies PGI Compiler 15.7, PGI OpenMPI 1.8.8, PGI HDF5 1.8.15-patch1

URL http://www.unidata.ucar.edu/software/netcdf/

Documentation http://www.unidata.ucar.edu/software/netcdf/docs/

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation,access, and sharing of array-oriented scientific data. NetCDF was developed and is maintained at Unidata. Unidataprovides data and software tools for use in geoscience education and research. Unidata is part of the UniversityCorporation for Atmospheric Research (UCAR) Community Programs (UCP). Unidata is funded primarily by theNational Science Foundation.

Usage

This version of NetCDF was compiled using version 15.7 of the PGI Compiler, OpenMPI 1.8.8 and HDF5 1.8.15-patch1. To make it available, run the following module command after starting a qsh or qrsh session

module load libs/pgi/netcdf/4.3.3.1

This module also loads the relevant modules for OpenMPI, PGI Compiler and HDF5. To see which modules havebeen loaded, use the command module list

Installation notes


Version 4.3.3.1

Compiled using PGI 15.7, OpenMPI 1.8.8 and HDF5 1.8.15-patch1

• install_pgi_netcdf_4.3.3.1.sh Install script

• Install logs are on the system at /usr/local/packages6/libs/pgi/netcdf/4.3.3.1/install_logs

• Modulefile located on the system at ls /usr/local/modulefiles/libs/pgi/netcdf/4.3.3.1


http://www.unidata.ucar.edu/software/netcdf/

http://www.unidata.ucar.edu/software/netcdf/docs/


OpenBLAS

OpenBLAS


URL http://www.openblas.net/

OpenBLAS is a library that provides low-level C and Fortran routines for linear algebra. It provides optimised versionsof the routines described in the BLAS specification. It is based on GotoBLAS2 1.13 BSD version.

Usage

To make OpenBLAS’s shared libraries, static libraries and header files available you need to run:

module load libs/gcc/4.9.2/openblas/0.2.19

Note: this will also load GCC 4.9.2 as OpenBLAS depends on the libgfortran, libquadmath and libgcc_slibraries provided by that compiler.

You can then statically link with libopenblas.a or with -lopenblas if you want to use the shared library.

The library is multithreaded. You can the number of threads with using one of the following environment variables:

* `èxport OPENBLAS_NUM_THREADS=4``

* `èxport GOTO_NUM_THREADS=4``

* `èxport OMP_NUM_THREADS=4``

The priorities are OPENBLAS_NUM_THREADS > GOTO_NUM_THREADS > OMP_NUM_THREADS.

Note: The build process console output indicates that the maximum number of threads is 12.

The library was compiled on and optimised for the Intel Nehalem architecture (inc. Intel Xeon X5650 CPUs) butshould also run on the newer Intel Sandy Bridge-EP archtecture (Intel Xeon E5 2650/2660/2670 CPUs).

Documentation

See the project website including the FAQ.

Installation notes


Version 0.2.19

This was compiled with GCC 4.9.2 on an Intel Xeon X5650 CPU.

1. First, download, configure, build, test and install using this script.

2. The default makefile target runs a test suite. All tests should pass (see the build process consoleoutput.

2.2. Iceberg 209

http://www.openblas.net/

https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms

https://github.com/xianyi/OpenBLAS/wiki/Faq


3. Next, install this modulefile as /usr/local/modulefiles/libs/gcc/4.9.2/openblas/0.2.19

pcre

pcre

Version 8.37


Dependancies None

URL http://www.pcre.org/

Location /usr/local/packages6/libs/gcc/4.4.7/pcre/8.37

The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax andsemantics as Perl 5. PCRE has its own native API, as well as a set of wrapper functions that correspond to the POSIXregular expression API. The PCRE library is free, even for building proprietary software.

Usage

To make this library available, run the following module command after starting a qsh or qrsh session.

module load libs/gcc/4.4.7/pcre/8.37

This also makes the updated pcregrep command available and will replace the system version. Check the versionyou are using with pcregrep -V

pcregrep -V

pcregrep version 8.37 2015-04-28

Installation notes


qrshtar -xvzf pcre-8.37.tar.gzcd pcre-8.37mkdir -p /usr/local/packages6/libs/gcc/4.4.7/pcre/8.37./configure --prefix=/usr/local/packages6/libs/gcc/4.4.7/pcre/8.37

The configuration details were

pcre-8.37 configuration summary:

Install prefix .................. : /usr/local/packages6/libs/gcc/4.4.7/pcre/8.37C preprocessor .................. : gcc -EC compiler ...................... : gccC++ preprocessor ................ : g++ -EC++ compiler .................... : g++


http://www.pcre.org/


Linker .......................... : /usr/bin/ld -m elf_x86_64C preprocessor flags ............ :C compiler flags ................ : -g -O2 -fvisibility=hiddenC++ compiler flags .............. : -O2 -fvisibility=hidden -fvisibility-inlines-→˓hiddenLinker flags .................... :Extra libraries ................. :

Build 8 bit pcre library ........ : yesBuild 16 bit pcre library ....... : noBuild 32 bit pcre library ....... : noBuild C++ library ............... : yesEnable JIT compiling support .... : noEnable UTF-8/16/32 support ...... : noUnicode properties .............. : noNewline char/sequence ........... : lf\R matches only ANYCRLF ......... : noEBCDIC coding ................... : noEBCDIC code for NL .............. : n/aRebuild char tables ............. : noUse stack recursion ............. : yesPOSIX mem threshold ............. : 10Internal link size .............. : 2Nested parentheses limit ........ : 250Match limit ..................... : 10000000Match limit recursion ........... : MATCH_LIMITBuild shared libs ............... : yesBuild static libs ............... : yesUse JIT in pcregrep ............. : noBuffer size for pcregrep ........ : 20480Link pcregrep with libz ......... : noLink pcregrep with libbz2 ....... : noLink pcretest with libedit ...... : noLink pcretest with libreadline .. : noValgrind support ................ : noCode coverage ................... : no


makemake install

Testing was performed and all tests passed

make check

============================================================================Testsuite summary for PCRE 8.37============================================================================# TOTAL: 5# PASS: 5# SKIP: 0# XFAIL: 0# FAIL: 0# XPASS: 0# ERROR: 0============================================================================

2.2. Iceberg 211


Module File

Module File Location: /usr/local/modulefiles/libs/gcc/4.4.7/pcre/8.37

#%Module1.0######################################################################### pcre 8.37 module file##


proc ModulesHelp { } {puts stderr "Makes the pcre 8.37 library available"

}

module-whatis "Makes the pcre 8.37 library available"

set PCRE_DIR /usr/local/packages6/libs/gcc/4.4.7/pcre/8.37

prepend-path LD_LIBRARY_PATH $PCRE_DIR/libprepend-path CPATH $PCRE_DIR/includeprepend-path PATH $PCRE_DIR/bin

PROJ.4

PROJ.4


URL https://github.com/OSGeo/proj.4

PROJ.4 consists of programs and a library for managing cartographic projections.

Usage

By running

$ module load libs/gcc/6.2/proj/4.9.3

you

• add several PROJ.4 programs to your PATH environment variable

• allow other programs to make use of (dynamically link against) the PROJ.4 library

You can run proj to test that you are running the required version

$ projRel. 4.9.3, 15 August 2016usage: proj [ -bCeEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]


https://github.com/OSGeo/proj.4


Documentation

Standard man pages are available for the provided commands/functions cs2cs, geod, proj, geodesic andpj_init.


$ man proj

Installation notes


Version 4.9.3

PROJ.4 4.9.3 was compiled with v6.2.0 of the GCC compiler suite.


2. Download, build and install PROJ.4 using :download‘this script (install.sh) </ice-berg/software/install_scripts/libs/gcc/6.2/proj/4.9.3/install.sh>‘. Files are installed into /usr/local/packages6/libs/gcc/6.2/proj/4.9.3/

3. Install this modulefile as /usr/local/modulefiles/libs/gcc/6.2/proj/4.9.3

SCOTCH

SCOTCH


URL http://www.labri.fr/perso/pelegrin/scotch/

Location /usr/local/packages6/libs/gcc/5.2/scotch/6.0.4

Software package and libraries for sequential and parallel graph partitioning, static mapping and clustering, sequentialmesh and hypergraph partitioning, and sequential and parallel sparse matrix block ordering.

Usage


module load libs/gcc/5.2/scotch/6.0.4

Installation notes

This section is primarily for administrators of the system. SCOTCH 6.0.4 was compiled with gcc 5.2 using the bashfile install_scotch.sh

2.2. Iceberg 213

http://www.labri.fr/perso/pelegrin/scotch/


SimBody

SimBody

Version 3.5.3


Dependancies compilers/gcc/4.8.2

URL https://simtk.org/home/simbody

Location /usr/local/packages6/libs/gcc/4.8.2/simbody/3.5.3

Usage


module load libs/gcc/4.8.2/simbody

Installing


Installed using the following procedure:


cmake ../simbody-Simbody-3.5.3/ -DCMAKE_INSTALL_PREFIX=/usr/local/packages6/libs/gcc/→˓4.8.2/simbody/3.5.3

make -j 8

wxWidgets

wxWidgets

Version 3.1.0

URL http://www.wxwidgets.org/

“wxWidgets is a C++ library that lets developers create applications for Windows, Mac OS X, Linux and other plat-forms with a single code base. It has popular language bindings for Python, Perl, Ruby and many other languages,and unlike other cross-platform toolkits, wxWidgets gives applications a truly native look and feel because it uses theplatform’s native API rather than emulating the GUI. It’s also extensive, free, open-source and mature.”

Usage

Activate wxWidgets using:


https://simtk.org/home/simbody

http://www.wxwidgets.org/


module load libs/gcc/4.9.2/wxWidgets/3.1.0

You can then check that wxWidgets is available using:

$ wx-config --version3.1.0

Installation notes

This section is primarily for system administrators.

Version 3.1.0

This is a pre-requisite for ctffind 4.x. It was built using gcc 4.9.2. Unicode support was enabled.

1. Download, configure, compile and install using this script

2. Install this modulefile as /usr/local/modulefiles//libs/gcc/4.9.2/wxWidgets/3.1.0

zlib

zlib

Version 1.2.8

URL http://zlib.net/

Location /usr/local/packages6/libs/gcc/4.4.7/zlib/1.2.8

A Massively Spiffy Yet Delicately Unobtrusive Compression Library.

Usage


module load libs/gcc/4.4.7/zlib/1.2.8

Installation notes


It was built with gcc 4.4.7

#!/bin/bash

install_dir=/usr/local/packages6/libs/gcc/4.4.7/zlib/1.2.8

wget http://zlib.net/zlib-1.2.8.tar.gztar -xvzf ./zlib-1.2.8.tar.gzcd zlib-1.2.8

2.2. Iceberg 215

http://zlib.net/


mkdir -p $install_dir

./configure --prefix=$install_dirmakemake install

testing

The library was tested with make check. The results were

make check

hello worldzlib version 1.2.8 = 0x1280, compile flags = 0xa9uncompress(): hello, hello!gzread(): hello, hello!gzgets() after gzseek: hello!inflate(): hello, hello!large_inflate(): OKafter inflateSync(): hello, hello!inflate with dictionary: hello, hello!

*** zlib test OK ***hello worldzlib version 1.2.8 = 0x1280, compile flags = 0xa9uncompress(): hello, hello!gzread(): hello, hello!gzgets() after gzseek: hello!inflate(): hello, hello!large_inflate(): OKafter inflateSync(): hello, hello!inflate with dictionary: hello, hello!

*** zlib shared test OK ***hello worldzlib version 1.2.8 = 0x1280, compile flags = 0xa9uncompress(): hello, hello!gzread(): hello, hello!gzgets() after gzseek: hello!inflate(): hello, hello!large_inflate(): OKafter inflateSync(): hello, hello!inflate with dictionary: hello, hello!

*** zlib 64-bit test OK ***

Module File

Module location is /usr/local/modulefiles/libs/gcc/4.4.7/zlib/1.2.8. Module contents

#%Module1.0######################################################################### zlib 1.2.8 module file##




##

proc ModulesHelp { } {puts stderr "Makes the zlib 1.2.8 library available"

}

module-whatis "Makes the zlib 1.2.8 library available"

set ZLIB_DIR /usr/local/packages6/libs/gcc/4.4.7/zlib/1.2.8

prepend-path LD_LIBRARY_PATH $ZLIB_DIR/libprepend-path CPATH $ZLIB_DIR/includeprepend-path MANPATH $ZLIB_DIR/share/man

Development Tools on Iceberg

CMake

CMake is a build tool commonly used when compiling other libraries.

Usage

CMake can be loaded with:

module load compilers/cmake/3.3.0

Usage often involves:

1. Creating and cd-ing into a dedicated build directory within a source tree then

2. Running something like cmake -DSOME_OPTION -DANOTHER_OPTION ..

Installation

Run the following commands:

module load apps/python/2.7

./bootstrap --prefix=/usr/local/packages6/cmake/3.3.0/--mandir=/usr/local/packages6/cmake/3.3.0/man --sphinx-man

gmake -j8

gmake install

Module file:

module-whatis loads the necessary `cmake-3.3.0' library pathsprepend-path PATH /usr/local/packages6/cmake/3.3.0/binprepend-path MANPATH /usr/local/packages6/cmake/3.3.0/man

2.2. Iceberg 217


GNU Compiler Collection (gcc)

The GNU Compiler Collection (gcc) is a widely used, free collection of compilers including C (gcc), C++ (g++) andFortran (gfortran). The defaut version of gcc on the system is 4.4.7

gcc -v

Using built-in specs.Target: x86_64-redhat-linuxConfigured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/→˓share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --→˓enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --→˓enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --→˓enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --→˓disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-→˓libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/→˓eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-→˓tune=generic --with-arch_32=i686 --build=x86_64-redhat-linuxThread model: posixgcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)

It is possible to switch to other versions of the gcc compiler suite using modules. After connecting to iceberg (seeEstablishing a SSH connection), start an interactive sesssion with the qrsh or qsh command. Choose the version ofthe compiler you wish to use using one of the following commands

module load compilers/gcc/6.2module load compilers/gcc/5.4module load compilers/gcc/5.3module load compilers/gcc/5.2module load compilers/gcc/4.9.2module load compilers/gcc/4.8.2module load compilers/gcc/4.5.3

Alternatively load the most recent available version using

module load compilers/gcc

Confirm that you’ve loaded the version of gcc you wanted using gcc -v.

Documentation

man pages are available on the system. Once you have loaded the required version of gcc, type

man gcc

• What’s new in the gcc version 6 series?


Installation Notes

These notes are primarily for system administrators

• gcc version 6.2 was installed using :

– install_gcc_6.2.sh


https://gcc.gnu.org/gcc-6/changes.html



– gcc 6.2 modulefile located on the system at /usr/local/modulefiles/compilers/gcc/6.2




• Installation notes for version 5.3 are not available.




• gcc version 4.9.2 was installed using :

– install_gcc_4.9.2.sh

– gcc 4.9.2 modulefile located on the system at /usr/local/modulefiles/compilers/gcc/4.9.2

• Installation notes for versions 4.8.2 and below are not available.

git

git

Latest version 2.5

Dependancies gcc 5.2

URL https://git-scm.com/

Git is a free and open source distributed version control system designed to handle everything from small to very largeprojects with speed and efficiency.

Usage

An old version of git is installed as part of the system’s opertaing system. As such, it is available everywhere, includingon the log-in nodes

$ git --versiongit version 1.7.1

This was released in April 2010. We recommend that you load the most up to date version using modules - somethingthat can only be done after starting an interactive qrsh or qsh session

module load apps/gcc/5.2/git/2.5

2.2. Iceberg 219

https://git-scm.com/


Installation notes

Version 2.5 of git was installed using gcc 5.2 using the following install script and module file:

• install_git_2.5.sh

• git 2.5 modulefile located on the system at /usr/local/modulefiles/compilers/git/2.5

Intel Compilers

The Intel compilers help create C, C++ and Fortran applications that can take full advantage of the advanced hardwarecapabilities available in Intel processors and co-processors. They also simplify that development by providing highlevel parallel models and built-in features like explicit vectorization and optimization reports.

Several versions of the Intel compilers were installed on this cluster as part of Intel Parallel Studio but can be usedwith or without the other Parallel Studio components.

Usage and compilation examples

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. To make one of the versions of the Intel compilers available, run one of the following module commands

module load compilers/intel/17.0.0module load compilers/intel/15.0.3module load compilers/intel/14.0module load compilers/intel/12.1.15module load compilers/intel/11.0

C

To compile the C hello world example into an executable called hello using the Intel C compiler:

icc hello.c -o hello

C++

To compile the C++ hello world example into an executable called hello using the Intel C++ compiler:

icpc hello.cpp -o hello

Fortran

To compile the Fortran hello world example into an executable called hello using the Intel Fortran compiler:

ifort hello.f90 -o hello

Detailed Documentation

Once you have loaded the module on Iceberg, man pages are available for Intel compiler products

man ifortman icc

The following links are to Intel’s website:



• User and Reference Guide for the Intel® C++ Compiler 17.0

• User and Reference Guide for the Intel® Fortran Compiler 17.0

• Step by Step optimizing with Intel C++ Compiler



Related Software on the system

TODO: add link to NAG Fortran Library (serial)

Installation Notes


Version 17.0.0


This modulefile was installed as /usr/local/modulefiles/compilers/intel/17.0.0.

Version 15.0.3

• The install is located on the system at /usr/local/packages6/compilers/intel/2015/

• The license file is at /usr/local/packages6/compilers/intel/license.lic

• The environment variable INTEL_LICENSE_FILE is set by the environment module and points to the licensefile location

• Download the files l_ccompxe_2015.3.187.tgz (C/C++) and l_fcompxe_2015.3.187.tgz (For-tran) from Intel Portal.

• Put the above .tgz files in the same directory as install_intel15.sh :down-load:</iceberg/software/install_scripts/compilers/intel/2015.3/install_intel15.sh> and silent_master.cfg

• Run install_intel15.sh

• To find what was required in the module file:

env > base.envsource /usr/local/packages6/compilers/intel/2015/composer_xe_2015.3.187/bin/→˓compilervars.sh intel64env > after_intel.envdiff base.env after_intel.env

• The module file is on iceberg at /usr/local/modulefiles/compilers/intel/15.0.3

2.2. Iceberg 221

https://software.intel.com/en-us/intel-cplusplus-compiler-17.0-user-and-reference-guide-intel-system-studio-2017

https://software.intel.com/en-us/intel-fortran-compiler-17.0-user-and-reference-guide

https://software.intel.com/en-us/articles/step-by-step-optimizing-with-intel-c-compiler


Versions 14 and below

Installation notes are not available for these older versions of the Intel compilers.

Intel Parallel Studio

Intel Parallel Studio XE is a software development suite that helps boost application performance by taking advantageof the ever-increasing processor core count and vector register width available in Intel Xeon processors and othercompatible processors. Its core components are compilers and libraries for fast linear algebra, data transformation andparallelisation.

Composer Edition 2017

This includes:

• Intel C++ and Fortran compilers: these help create C, C++ and Fortran applications that “can take full advan-tage of the advanced hardware capabilities available in Intel processors and co-processors. They also simplifythat development by providing high level parallel models and built-in features like explicit vectorization andoptimization reports”.

• Data Analytics Acceleration Library (DAAL): functions for data analysis (characterization, summarization, andtransformation) and Machine Learning (regression, classification, and more).

• Math Kernel Library (MKL): This library provides highly optimized, threaded and vectorized functions to max-imize performance on each processor family. Utilizes de facto standard C and Fortran APIs for compatibilitywith BLAS, LAPACK and FFTW functions from other math libraries.

• Integrated Performance Primitives (IPP): “high-quality, production-ready, low-level building blocks for imageprocessing, signal processing, and data processing (data compression/decompression and cryptography) appli-cations.”

• Threading Building Blocks (TBB): lets you write “parallel C++ programs that take full advantage of multicoreperformance, that are portable and composable, and that have future-proof scalability.”

It does not include Intel’s MPI implementation. See Intel’s site for further details of what Parallel Studio ComposerEdition includes.


Some components of Parallel Studio are freely available for those only wanting community support; other components,such as the compilers are commercial software.

The University has purchased a number of Intel-supported licenses for the Parallel Studio Composer Edition productslisted above. These can be accessed by (and are shared between) the University’s HPC clusters and researchers’ ownmachines.

Installation Notes


Parallel Studio XE Composer Edition 2017.0


https://software.intel.com/en-us/intel-parallel-studio-xe/details


1. Download parallel_studio_xe_2017_composer_edition.tgz from the Intel Portal, save in /usr/local/media/protected/intel/2017.0/ then make it only readable by the app-adminsgroup.

2. Ensure details of the Intel license server are in the file /usr/local/packages6/compilers/intel/license.lic

3. Run this script. This installs Parallel Studio into /usr/local/packages6/compilers/intel-pe-xe-ce/2017.0/binary/. Products are activated using the aforementioned license file duringthe installation process.

4. Install several modulefiles to locations under /usr/local/modulefiles. These modulefiles set theINTEL_LICENSE_FILE environment variable to the location of the aforementioned license file and set otherenvironment variables required by the different Parallel Studio products. There is one modulefile for all ParallelStudio software and other modulefiles for specific products.

• The Compilers modulefile should be installed as /usr/local/modulefiles/compilers/intel/17.0.0.

• The DAAL modulefile should be installed as /usr/local/modulefiles/libs/binlibs/intel-daal/2017.0.

• The IPP modulefile should be installed as /usr/local/modulefiles/libs/binlibs/intel-ipp/2017.0.

• The MKL modulefile should be installed as /usr/local/modulefiles/libs/binlibs/intel-mkl/2017.0.

• The TBB modulefile should be installed as /usr/local/modulefiles/libs/binlibs/intel-tbb/2017.0.

• See the (TCL) modulefiles for details of how they were derived from Intel-supplied environment-manipulating shell scripts.

5. Check that licensing is working by activating the Intel Compilers modulefile then try compiling a trivial Cprogram using the icc compiler.

NAG Fortran Compiler

The NAG Fortran Compiler is robust, highly tested, and valued by developers all over the globe for its checking capa-bilities and detailed error reporting. The Compiler is available on Unix platforms as well as for Microsoft Windowsand Apple Mac platforms. Release 6.0 has extensive support for both legacy and modern Fortran features, and alsosupports parallel programming with OpenMP.

Making the NAG Compiler available

After connecting to iceberg (see Establishing a SSH connection), start an interactive sesssion with the qsh or qrshcommand. To make a version of the NAG Fortran Compiler available, run one of following module commands:

module load compilers/NAG/6.1module load compilers/NAG/6.0

Compilation examples

To compile the Fortran hello world example into an executable called hello using the NAG compiler

2.2. Iceberg 223

https://en.wikipedia.org/wiki/%22Hello,_World!%22_program



nagfor hello.f90 -o hello


Once you’ve run one of the NAG Compiler module command, man documentation is available:

man nagfor

Online documentation:

• PDF version of the NAG Fortran Compiler Manual

• NAG Fortran Compiler Documentation Index (NAG’s Website)

• Differences between different versions can be found in the Release Notes.

Installation Notes

The following notes are primarily for system administrators:

Licensing

Add the license key(s) to /usr/local/packages5/nag/license.lic.

The license key needs to be updated annually before 31st July.

The NAG compiler environment modules (see below) set the environment variable $NAG_KUSARI_FILE to the pathto this file.

Version 6.1

1. Perform an unattended install using this script. The software will be installed into /usr/local/packages6/compilers/NAG/6.1.

2. Install this modulefile as /usr/local/modulefiles/compilers/NAG/6.1

3. Test the installation by compiling and building a sample Fortran 90 program

module load compilers/NAG/6.1nagfor -o /tmp/f90_util /usr/local/packages6/compilers/NAG/6.1/lib/NAG_Fortran/→˓f90_util.f90/tmp/f90_util

Version 6.0

Run the following interactively:

mkdir -p /usr/local/packages6/compilers/NAG/6.0tar -xvzf ./npl6a60na_amd64.tgzcd NAG_Fortran-amd64/

Run the interactive install script

./INSTALL.sh

Accept the license and answer the questions as follows:


http://www.nag.co.uk/nagware/np/r60_doc/np60_manual.pdf

http://www.nag.co.uk/nagware/np.asp

https://www.nag.co.uk/nag-compiler


Install compiler binaries to where [/usr/local/bin]?/usr/local/packages6/compilers/NAG/6.0

Install compiler library files to where [/usr/local/lib/NAG_Fortran]?/usr/local/packages6/compilers/NAG/6.0/lib/NAG_Fortran

Install compiler man page to which directory [/usr/local/man/man1]?/usr/local/packages6/compilers/NAG/6.0/man/man1

Suffix for compiler man page [1] (i.e. nagfor.1)?Press enter

Install module man pages to which directory [/usr/local/man/man3]?/usr/local/packages6/compilers/NAG/6.0/man/man3

Suffix for module man pages [3] (i.e. f90_gc.3)?Press Enter

Install this modulefile as /usr/local/modulefiles/compilers/NAG/6.0

Finally, test the installation by compiling and building a sample Fortran 90 program

module load compilers/NAG/6.0nagfor -o /tmp/f90_util /usr/local/packages6/compilers/NAG/6.0/lib/NAG_Fortran/f90_→˓util.f90/tmp/f90_util

NVCC - Nvidia CUDA Compiler

To compile GPU code using the NVIDIA compiler, nvcc, first start an interactive session (see Interactive use of theGPUs). Next, you need to set up the compiler environment via one of the following module statements:

module load libs/cuda/8.0.44module load libs/cuda/7.5.18module load libs/cuda/6.5.14module load libs/cuda/4.0.17module load libs/cuda/3.2.16

depending on the version of CUDA you intend to use. This makes the nvcc CUDA compiler available.



• less than 5.0.0

It is therefore recommended that you load the most recent 4.x version of GCC when building CUDA programs onIceberg:


An example of the use of nvcc:

nvcc filename.cu -arch sm_20

2.2. Iceberg 225


will compile the CUDA program contained in the file filename.cu. The -arch flag above signifies the computecapability of the intended GPU hardware, which is 2.0 for our current GPU modules. If you are intending to generatecode for older architectures you may have to specify sm_10 or sm_13 for example.

PGI Compilers

The PGI Compiler suite offers C,C++ and Fortran Compilers. For full details of the features of this compiler suite, seePGI’s website.

Making the PGI Compilers available

After connecting to iceberg (see Establishing a SSH connection), start an interactive session with the qsh or qrshcommand. To make one of the versions of the PGI Compiler Suite available, run one of the following module com-mands

module load compilers/pgi/16.10module load compilers/pgi/15.10module load compilers/pgi/15.7module load compilers/pgi/14.4module load compilers/pgi/13.1

Once you’ve loaded the module, you can check the version with

pgcc --version


C

To compile a C hello world example into an executable called hello using the PGI C compiler

pgcc hello.c -o hello

C++

To compile a C++ hello world example into an executable called hello using the PGI C++ compiler

pgc++ hello.cpp -o hello

Fortran

To compile a Fortran hello world example into an executable called hello using the PGI Fortran compiler

pgf90 hello.f90 -o hello

Compiling on the GPU using the PGI Compiler

Start an interctive GPU session (Interactive use of the GPUs) and run one of the following module commands, de-pending on which version of the compilers you wish to use


http://www.pgroup.com/products/pgiworkstation.htm


module load compilers/pgi/16.10module load compilers/pgi/15.10module load compilers/pgi/15.7module load compilers/pgi/14.4module load compilers/pgi/13.1

The PGI compilers have several features that make them interesting to users of GPU hardware:-

OpenACC Directives

OpenACC is a relatively new way of programming GPUs that can be significantly simpler to use than low-levellanguage extensions such as CUDA or OpenCL. From the OpenACC website :

The OpenACC Application Program Interface describes a collection of compiler directives to specifyloops and regions of code in standard C, C++ and Fortran to be offloaded from a host CPU to an attachedaccelerator. OpenACC is designed for portability across operating systems, host CPUs, and a wide rangeof accelerators, including APUs, GPUs, and many-core coprocessors.

The directives and programming model defined in the OpenACC API document allow programmers tocreate high-level host+accelerator programs without the need to explicitly initialize the accelerator, man-age data or program transfers between the host and accelerator, or initiate accelerator startup and shut-down.

For more details concerning OpenACC using the PGI compilers, see The PGI OpenACC website.

CUDA Fortran

In mid 2009, PGI and NVIDIA cooperated to develop CUDA Fortran. CUDA Fortran includes a Fortran 2003 compilerand tool chain for programming NVIDIA GPUs using Fortran.

• CUDA Fortran Programming Guide.

CUDA-x86

NVIDIA CUDA was developed to enable offloading computationally intensive kernels to massively parallel GPUs.Through API function calls and language extensions, CUDA gives developers explicit control over the mapping ofgeneral-purpose computational kernels to GPUs, as well as the placement and movement of data between an x86processor and the GPU.

The PGI CUDA C/C++ compiler for x86 platforms allows developers using CUDA to compile and optimize theirCUDA applications to run on x86-based workstations, servers and clusters with or without an NVIDIA GPU accelera-tor. When run on x86-based systems without a GPU, PGI CUDA C applications use multiple cores and the streamingSIMD (Single Instruction Multiple Data) capabilities of Intel and AMD CPUs for parallel execution.

• PGI CUDA-x86 guide.

Installation Notes

Version 16.10

The interactive installer was slightly different to that of 15.7 (below) but the questions and answers were essentiallythe same.

Version 15.10

2.2. Iceberg 227

http://www.openacc-standard.org/About_OpenACC

http://www.pgroup.com/resources/accel.htm

http://www.pgroup.com/lit/whitepapers/pgicudaforug.pdf

http://www.pgroup.com/resources/cuda-x86.htm


The interactive installer was slightly different to that of 15.7 (below) but the questions and answers were essentiallythe same.

Version 15.7

The installer is interactive. Most of the questions are obvious. Here is how I answered the rest

Installation type

A network installation will save disk space by having only one copy of thecompilers and most of the libraries for all compilers on the network, andthe main installation needs to be done once for all systems on the network.

1 Single system install2 Network install

Please choose install option: 1

Path

Please specify the directory path under which the software will be installed.The default directory is /opt/pgi, but you may install anywhere you wish,assuming you have permission to do so.

Installation directory? [/opt/pgi] /usr/local/packages6/compilers/pgi

CUDA and AMD components

Install CUDA Toolkit Components? (y/n) yInstall AMD software components? (y/n) y

AMCL version

This PGI version links with ACML 5.3.0 by default. Also available:(1) ACML 5.3.0(2) ACML 5.3.0 using FMA4

Enter another value to override the default (1)1

Other questions

Install JAVA JRE [yes] yesInstall OpenACC Unified Memory Evaluation package? (y/n) nDo you wish to update/create links in the 2015 directory? (y/n) yDo you wish to install MPICH? (y/n) yDo you wish to generate license keys? (y/n) nDo you want the files in the install directory to be read-only? (y/n) n

The license file is on the system at /usr/local/packages6/compilers/pgi/license.dat and is a 5 seatnetwork license. Licenses are only used at compile time.

Extra install steps

Unlike gcc, the PGI Compilers do not recognise the environment variable LIBRARY_PATH which is used by a lotof installers to specify the locations of libraries at compile time. This is fixed by creating a siterc file at /usr/local/packages6/compilers/pgi/linux86-64/VER/bin/siterc with the following contents



# get the value of the environment variable LIBRARY_PATHvariable LIBRARY_PATH is environment(LD_LIBRARY_PATH);variable inc_path is environment(CPATH);

# split this value at colons, separate by -L, prepend 1st one by -Lvariable library_path isdefault($if($LIBRARY_PATH,-L$replace($LIBRARY_PATH,":", -L)));

# add the -L arguments to the link lineappend LDLIBARGS=$library_path;append SITEINC=$inc_path;

Where VER is the version number in question: 15.7, 15.10 etc

At the time of writing (August 2015), this is documented on PGI’s website.

Modulefile

Version 15.10 The PGI compiler installer creates a suitable modulefile that’s configured to our system. It puts it at /usr/local/packages6/compilers/pgi/modulefiles/pgi64/15.10 so all that is required is to copythis to where we keep modules at /usr/local/modulefiles/compilers/pgi/15.10

Version 15.7

The PGI compiler installer creates a suitable modulefile that’s configured to our system. It puts it at /usr/local/packages6/compilers/pgi/modulefiles/pgi64/15.7 so all that is required is to copy this to where wekeep modules at /usr/local/modulefiles/compilers/pgi/15.7

MPI

OpenMPI (gcc version)



Dependancies gcc

URL http://www.open-mpi.org/

The Open MPI Project is an open source Message Passing Interface implementation that is developed and maintainedby a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise,technologies, and resources from all across the High Performance Computing community in order to build the bestMPI library available. Open MPI offers advantages for system and software vendors, application developers andcomputer science researchers.

Module files

The latest version is made available using

module load mpi/gcc/openmpi

2.2. Iceberg 229

https://www.pgroup.com/support/link.htm#lib_path_ldflags

http://www.open-mpi.org/


Alternatively, you can load a specific version using one of

module load mpi/gcc/openmpi/1.10.1module load mpi/gcc/openmpi/1.10.0module load mpi/gcc/openmpi/1.8.8module load mpi/gcc/openmpi/1.8.3module load mpi/gcc/openmpi/1.6.4module load mpi/gcc/openmpi/1.4.4module load mpi/gcc/openmpi/1.4.3

Installation notes


Version 1.10.1

Compiled using gcc 4.4.7.

• install_openMPI_1.10.1.sh Downloads, compiles and installs OpenMPI 1.10.0 using the system gcc.

• Modulefile 1.10.1 located on the system at /usr/local/modulefiles/mpi/gcc/openmpi/1.10.1

Version 1.10.0


• install_openMPI_1.10.0.sh Downloads, compiles and installs OpenMPI 1.10.0 using the system gcc.

• Modulefile 1.10 located on the system at /usr/local/modulefiles/mpi/gcc/openmpi/1.10.0

Version 1.8.8


• install_openMPI.sh Downloads, compiles and installs OpenMPI 1.8.8 using the system gcc.

• Modulefile located on the system at /usr/local/modulefiles/mpi/gcc/openmpi/1.8.8

OpenMPI (PGI version)



Support Level FULL

Dependancies PGI



These versions of OpenMPI make use of the PGI compiler suite.




Module files

The latest version is made available using

module load mpi/pgi/openmpi

Alternatively, you can load a specific version using one of

module load mpi/pgi/openmpi/1.8.8module load mpi/pgi/openmpi/1.6.4

Installation notes


Version 1.8.8

Compiled using PGI 15.7.

• install_openMPI.sh downloads, compiles and installs OpenMPI 1.8.8 using v15.7 of the PGI Compiler.

• Modulefile located on the system at /usr/local/modulefiles/mpi/pgi/openmpi/1.8.8

Installation notes for older versions are not available.

Accessing Iceberg

Note: If you have not connected to iceberg before the Using the HPC Systems documentation will guide you throughthe process.

Below is a list of the various ways to connect to or use iceberg:

Hardware-accelerated graphics rendering

Software that requires OpenGL and a graphics card to render complex visualisations cannot be run on all of Iceberg’snodes as:

• Only a limited number of nodes have graphics cards

• None of the nodes that do have a monitor attached

To run such software on Iceberg we therefore need:

• A means to automate the finding of a node with a graphics card that we can use for visualisation purposes

• A way of streaming the images rendered on a node through to the machine the user is sat in front of.

We have set up on Iceberg a system for hardware-accelerated visualisation that uses two main tools: VirtualGL andTigerVNC. See below for simple usage instructions.

Usage instructions

1. Download and install TigerVNC on your machine. TigerVNC is a program that allows you to efficiently viewgraphical programs on one computer that are actually running on another computer.

2.2. Iceberg 231

http://sourceforge.net/projects/tigervnc/


2. Connect to Iceberg

3. On the login node (not a worker node) run the command qsh-vis. The output should look something like thefollowing:

[te1st@iceberg-login1 ~]$ qsh-vis

NOTE: you can only run 1 GPU acclerated session

New 'node000:1 (te1st)' desktop is node000:1

Starting applications specified in /home/te1st/.vnc/xstartupLog file is /home/te1st/.vnc/node000:1.log

Accelerated VNC graphics session started

*******To connect: *******

Either startup a web-browser and enter the address:http://iceberg-gateway.shef.ac.uk:5801

or

use the TigerVNC application to connect to:iceberg-gateway.shef.ac.uk:5901

Make sure to enter your normal Iceberg username/password when prompted.

To end session : press [enter]

4. Leave that terminal running.

5. On your machine start the ‘VNC Viewer’ program that comes with TigerVNC (this is called vncviewer onLinux). You should then see a dialog box like this:

6. Enter the connection details for TigerVNC that were issued by the qsh-vis command e.g.iceberg-gateway.shef.ac.uk:5901 (NB the last four digits may differ when you run qsh-vis).

7. Click Connect

8. You should now see a desktop within a window. This desktop is running on a worker node (in the case of the



presented example this is node000`; see the qsh-vis output) that is equipped with a graphics card (Optional:run nvidia-smi to see what type of graphics card). A terminal window is automatically started from whichyou can load modules and start applications that require hardware-accelerated graphics.

9. When you are finished, close VNC Viewer then return to the terminal within which you started qsh-vis andpress enter to stop the worker session (and allow someone else to use that graphics-card-equipped node).

Resources available to qsh-vis sessions

• Sessions started using qsh-vis by default have allocated to them:

– 1 CPU core

– 1 GPU

• You can request additional resources by passing the same parameters to qsh-vis that can be used withqrsh/qrshx/qsh/qsub (see Starting interactive jobs and submitting batch jobs).

• Research groups who have purchased their own GPU visualisation nodes may have different defaults.

Availability

At present only one graphics-card-equipped node is available for use with qsh-vis so there may be contention forthis resource. Some research groups have purchased their own visualisation nodes.

Technical details

Behind the scenes:

2.2. Iceberg 233


• qsh-vis sets the default resources to be requested for the interactive session (based on whether the userbelongs to a research group that has dedicated visualisation nodes)...

• ...then uses qrsh to start a script with these resources.

• This script then starts a TigerVNC vncserver on a port that is unique over the range of machines on whichhardware-accelerated visualisation sessions can be started.

• TigerVNC supports VirtualGL, a means of streaming the images rendered by say a graphics card to a remotemachine.

• The aforementioned script then kills the created Xvnc process when Enter is pressed.

Iceberg specifications

• Total CPUs: 3440 cores

• Total GPUs: 16 units

• Total Memory: 31.8 TBytes

• Permanent Filestore: 45 TBytes

• Temporary Filestore: 260 TBytes

• Physical size: 8 Racks

• Maximum Power Consumption: 83.7 KWatts

• All nodes are connected via fast infiniband.

For reliability, there are two iceberg head-nodes ‘for logging in’ configured to take over from each other in case ofhardware failure.

Worker Nodes CPU Specifications

Intel Ivybridge based nodes

• 92 nodes, each with 16 cores and 64 GB of total memory (i.e. 4 GB per core).

• 4 nodes, each with 16 cores and 256 GB of total memory (i.e. 16GB per core).

• Each node uses2 of Intel E5 2650V2 8-core processors (hence 2*8=16 cores).

• Scratch space on local disk of on each node is 400 GB

Intel Westmere based nodes

• 103 nodes, each with 12 cores and 24 GB of total memory ( i.e. 2 GB per core )

• 4 nodes with 12 cores and 48 GB of total memory ( i.e. 4GB per core )

• Each node uses 2 of Intel X5650 6-core processors ( hence 2*6=12 cores )

GPU Units Specifications

8 Nvidia Tesla Kepler K40Ms GPU units

• Each GPU unit contains 2880 thread processor cores

• Each GPU unit has 12GB of GDR memory. Hence total GPU memory is 8*12=96 GB


http://www.virtualgl.org/About/Introduction


• Each GPU unit is capable of about 4.3TFlop of single precision floating point performance, or 1.4TFlops atdouble precision.

8 Nvidia Tesla Fermi M2070s GPU units

• Each GPU unit contains 448 thread processor cores

• Each GPU unit contains 6GB of GDR memory. Hence total GPU memory is 8*6=48 GB

• Each GPU unit is capable of about 1TFlop of single precision floating point performance, or 0.5TFlops at doubleprecision.

Software and Operating System

Users normally log into a head node and then use one (or more) of the worker nodes to run their jobs on. Schedulingof users’ jobs on the worker nodes are managed by the ‘Sun Grid Engine’ software. Jobs can be run interactively ( qsh) or submitted as batch jobs ( qsub ).

• The operating system is 64-bit Scientific Linux (which is Redhat based) on all nodes

• The Sun Grid Engine for batch and interactive job scheduling

• Many Applications, Compilers, Libraries and Parallel Processing Tools. See Software on iceberg

Using GPUs on Iceberg

Requesting access to GPU facilities

Public GPU nodes have now been made available to Iceberg and ShARC users, these can be be used withoutacquiring extra permission.

Research groups also have an option to purchase and add nodes to the cluster to be managed by CiCs. For these nodes(e.g. Nvidia DGX-1 [COM]), permission from the group leader is required for access.

The node owner always have highest priority on the job queue but as a condition for the management of addi-tional nodes on the cluster, the nodes are expected to be used as a resource for running short jobs during idletime. If you would like more information about having CiCs add and manage custom nodes, please contact [email protected].

Interactive use of the GPUs

Once you are included in the GPU project group you may start using the GPU enabled nodes interactively by typing:

qsh -l gpu=1

the -l gpu= parameter determines how many GPUs you are requesting. Currently, the maximum number of GPUsallowed per job is set to 4, i.e. you cannot exceed -l gpu=4. Most jobs will only make use of one GPU.

If your job requires selecting the type of GPU hardware, one of the following two optional parameters can be used tomake that choice

qsh -l gpu_arch=nvidia-m2070qsh -l gpu_arch=nvidia-k40m

Interactive sessions provide you with 2 Gigabytes of CPU RAM by default which is significantly less than the amountof GPU RAM available. This can lead to issues where your session has insufficient CPU RAM to transfer data to and

2.2. Iceberg 235




from the GPU. As such, it is recommended that you request enough CPU memory to communicate properly with theGPU

qsh -l gpu_arch=nvidia-m2070 -l rmem=7Gqsh -l gpu_arch=nvidia-k40m -l rmem=13G

The above will give you 1GB more CPU RAM than GPU RAM for each of the respective GPU architectures.

Submitting batch GPU jobs

To run batch jobs on gpu nodes, edit your jobfile to include a request for GPUs, e.g. for a single GPU

#!/bin/bash#$ -l gpu=1

You can also use the the gpu_arch discussed aboved to target a specific GPU model

#!/bin/bash#$ -l gpu_arch=nvidia-m2070

Iceberg GPU Resources

Hardware

Iceberg currently contains publicly accessible 16 GPU units:

• 8 Nvidia Tesla Kepler K40M GPU units. Each unit contains 2880 CUDA cores, 12GB of memory and is capableof up to 1.43 Teraflops of double precision compute power.

• 8 Nvidia Tesla Fermi M2070 GPU units. Each unit contains 448 CUDA cores, 6GB of memory and is capableof up to 515 Gigaflops of double precision compute power.

GPU-enabled Software

• Applications

– ANSYS

– Maple

– MATLAB

– Theano

• Libraries

– CUDA

– cuDNN

• Development Tools

– PGI Compilers

– NVCC - Nvidia CUDA Compiler



Training materials

• Introduction to CUDA by GPUComputing@Sheffield

ShARC

This is the documentation for ShARC (Sheffield Advanced Research Computer), The University of Sheffield’s newHigh Performance Computing System and the successor to Iceberg.

Software on ShARC

These pages list the software available on ShARC. If you notice an error or an omission, or wish to request newsoftware please create an issue on our github page

Applications on Sharc

ANTs

ANTs

Version 2.1.0-a

Dependancies gcc/4.9.4

URL https://github.com/stnava/ANTs.git

Documentation http://stnava.github.io/ANTs/

The ANTs framework provides open-source functionality for deformable image registration with small or large defor-mations. ANTs includes N4 bias correction, additional evaluation of multiple modalities and organ systems, univariateor multivariate image segmentation, tighter integration with the Insight ToolKit, a well evaluated cortical thicknesspipeline and visualization tools and integration with R. ANTs serves as both a core library for further algorithm devel-opment and also as a command-line application-oriented toolkit. ANTs also has a permissive software license that al-lows it to be employed freely by industry. ANTs enables diffeomorphic normalization with a variety of transformationmodels, optimal template construction, multiple types of diffeomorphisms, multivariate similarity metrics, diffusiontensor processing and warping, image segmentation with and with-out priors and measurement of cortical thicknessfrom probabilistic segmentations. The normalization tools, alone, provide a near limitless range of functionality andallow the user to develop customized objective functions.

Usage

ANTs can be activated using the module file:

module load apps/ANTs/2.1.0-a/gcc-4.9.4

Note: this ANTs version is 783 commits ahead of Release 2.1.0 (Commit 646459b build), which we have namedVersion 2.1.0-a

2.3. ShARC 237


https://github.com/rcgsheffield/sheffield_hpc/issues

https://github.com/stnava/ANTs.git

http://stnava.github.io/ANTs/


Test

1. Download and unpack the following brain image data using:

cd some_directorywget https://github.com/stnava/BasicBrainMapping/tarball/master/stnava-→˓BasicBrainMapping-e157693.tar.gztar zxvf stnava-BasicBrainMapping-e157693.tar.gzcd stnava-BasicBrainMapping-e157693

2. Create a batch job submission script called test.sge containing:

#!/bin/bash#$ -pe smp 2export OMP_NUM_THREADS=2#$ -l rmem=8G#$ -o Ants.out#$ -e Ants.err#$ -m eba -M [email protected]

module load apps/ANTsexport ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=2 # controls multi-threading

dim=3 # image dimensionalitynm=BBM_Lesion

MultiplyImages 3 ./data/IXI594-Guys-1089-T1.nii.gz data/neg_lesion.nii.gz data/→˓T1_lesioned.nii.gz

antsRegistrationSyNQuick.sh -n 2 -d 3 -f data/IXI/T_template2.nii.gz -m data/T1_→˓lesioned.nii.gz -t s -o ${nm}_diff -x data/neg_lesion.nii.gz

rm grid.nii.gz jacobian.nii.gz

CreateJacobianDeterminantImage 3 ${nm}_diff1Warp.nii.gz jacobian.nii.gz 1

CreateWarpedGridImage 3 ${nm}_diff1Warp.nii.gz grid.nii.gz 1x0x1 10x10x10 3x3x3

3. Submit the job using qsub test.sge

The registered image/s can be viewed using ITK-SNAP.

Installation notes

ANTs was compiled using the install_ANTs.sh script, the module file is gcc-4.9.4.

CASTEP

CASTEP

Versions 16.11 and 18.1

Dependencies Intel compilers, Intel MKL libraries and Open MPI



URL http://www.castep.org/

Documentation http://www.castep.org/CASTEP/Documentation

Licensing

Only licensed users of CASTEP are entitled to use it and license details are available on CASTEP’s website. Accessto CASTEP on the system is controlled using a Unix group. That is, only members of the castep group can accessand run the program. To be added to this group, you will need to contact [email protected] andprovide evidence of your eligibility to use CASTEP.

Interactive Usage

The serial version of CASTEP should be used for interactive usage. After connecting to ShARC (see Establishing aSSH connection), start an interactive session with the qrsh or qrshx command. Make the serial version of CASTEPavailable using one of the following commands:

module load apps/castep/16.11/intel-15.0.7module load apps/castep/18.1/intel-17.0.0

The CASTEP executable is called castep.serial so if you execute:

castep.serial

You should get the following:

Usage:castep <seedname> : Run files <seedname>.cell [and <seedname>.param]

" [-d|--dryrun] <seedanme> : Perform a dryrun calculation on files <seedname>.→˓cell" [-s|--search] <text> : print list of keywords with <text> match in

→˓description" [-v|--version] : print version information" [-h|--help] <keyword> : describe specific keyword in <>.cell or <>.param" " all : print list of all keywords" " basic : print list of basic-level keywords" " inter : print list of intermediate-level keywords" " expert : print list of expert-level keywords" " dummy : print list of dummy keywords

If, instead, you get:

-bash: castep.serial: command not found

It is probably because you are not a member of the castep group. See Licensing for details on how to be added tothis group.

Interactive usage is fine for small CASTEP jobs such as the Silicon example given in the CASTEP Basics and Bondingtutorial.

To run this example, you can do:

# Get the files, decompress them and enter the directory containing themwget http://www.castep.org/files/Si2.tgztar -xvzf ./Si2.tgz

2.3. ShARC 239

http://www.castep.org/

http://www.castep.org/CASTEP/Documentation

http://www.castep.org/CASTEP/GettingCASTEP



cd Si2

# Run the CASTEP job in serialcastep.serial Si2

# Read the output using the less commandless Si2.castep

CASTEP has a built in help system. To get more information on using castep use:

castep.serial -help

Alternatively you can search for help on a particular topic:

castep.serial -help search keyword

or list all of the input parameters:

castep.serial -help search all

Batch Submission - Parallel

The parallel version of CASTEP is called castep.mpi. To make the parallel environment available, use one of thefollowing commands:

module load apps/castep/16.11/intel-15.0.7-openmpi-2.0.1module load apps/castep/18.1/intel-17.0.0-openmpi-2.0.1

As an example of a parallel submission, we will calculate the bandstructure of graphite following the CASTEP BandStructure and DOS tutorial.

After connecting to ShARC (see Establishing a SSH connection), start an interactive session with the qrsh or qrshxcommand. Download and decompress the example input files with the commands:

wget http://www.castep.org/files/bandstructure.tgztar -xvzf ./bandstructure.tgz

Enter the directory containing the input files for graphite:

cd bandstructure/graphite/

Create a file called submit.sge that contains the following:

#!/bin/bash#$ -pe mpi 4 # Run the calculation on 4 CPU cores#$ -l rmem=4G # Request 4 GB of real memory per coremodule load apps/castep/16.11/intel-15.0.7-openmpi-2.0.1

mpirun castep.mpi graphite

Submit it to the system with the command:

qsub submit.sge

After the calculation has completed, get an overview of the calculation by looking at the file graphite.castep:





more graphite.castep

Installation Notes


Version 16.11

Serial (no MPI) and parallel (MPI) builds were compiled. Both builds were compiled with Intel compiler 15.0.7(including the Intel MKL 2015.7 for BLAS and FFT routines). The parallel build was compiled using OpenMPI 2.0.1.

Both builds were installed using this script. Note that this compiles both builds in /data/$USER as the builddirectory must be availble to all cluster nodes to allow for subsequent Testing of the parallel build. ~2.2 GB of freespace is required.

• The serial build modulefile was installed as /usr/local/modulefiles/apps/castep/16.11/intel-15.0.7

• The parallel build modulefile was installed as /usr/local/modulefiles/apps/castep/16.11/intel-15.0.7-openmpi-2.0.1

Version 18.1

Serial (no MPI) and parallel (MPI) builds were compiled. Both builds were compiled with Intel compiler 17.0.0(including the Intel MKL 2017.0 for BLAS and FFT routines). The parallel build was compiled using OpenMPI 2.0.1.

Both builds were installed using this script.

• The serial build modulefile was installed as /usr/local/modulefiles/apps/castep/18.1/intel-17.0.0

• The parallel build modulefile was installed as /usr/local/modulefiles/apps/castep/18.1/intel-17.0.0-openmpi-2.0.1

Testing

Version 16.11, serial build

The following script was submitted via qsub from the Test subdirectory of the build directory:

#!/bin/bash#$ -l rmem=10Gmodule load apps/castep/16.11/intel-15.0.7

cd /scratch/$USER/castep/16.11/intel-15.0.7/serial/Test../bin/testcode.py -q --total-processors=1 -e castep.serial -c simple -v -v -v

All 416 tests passed. Results can be found in castep_16_11_serial_sharc_build_tests.log.

2.3. ShARC 241


Version 16.11, parallel build


#!/bin/bash#$ -pe mpi 4#$ -l rmem=10Gmodule load apps/castep/16.11/intel-15.0.7-openmpi-2.0.1

../bin/testcode.py -q --total-processors=4 --processors=4 -e castep.parallel -c→˓simple -v -v -v

All 416 tests passed. Results can be found in castep_16_11_mpi4_sharc_build_tests.log.

Version 18.1, serial build

The following command was issued in the CASTEP-18.1 build directory during an interactive session with 16 GBof memory (qrsh -l rmem=16G):

make check

All 455 tests passed.

Version 18.1, parallel build


#!/bin/bash#$ -pe mpi 4#$ -l rmem=4G#$ -l h_rt=48:00:00

module load apps/castep/18.1/intel-17.0.0-openmpi-2.0.1

../bin/testcode.py -q --total-processors=4 --processors=4 -e castep.mpi -c simple -v→˓-v -v

All 455 tests passed.

FastQC

FastQC

Version 0.11.7

Dependency Java for GUI. Module loaded for Java 1.8.0_102.

URL https://www.bioinformatics.babraham.ac.uk/projects/fastqc/

Documentation https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/


https://www.bioinformatics.babraham.ac.uk/projects/fastqc/

https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/


FastQC aims to provide a simple way to do some quality control checks on raw sequence data coming from highthroughput sequencing pipelines. It provides a modular set of analyses which you can use to give a quick impressionof whether your data has any problems of which you should be aware before doing any further analysis.

Usage

FastQC version can be activated using the module file:

module load apps/FastQC/0.11.7/binary

Then run using the command fastqc.

Installation notes

FastQC was installed using the install_fastqc.sh script, the module file is 0.11.7.

R

R



R is a statistical computing language.

Interactive Usage

After connecting to ShARC, start an interactive session with the qrshx command.

The latest version of R can be loaded with

module load apps/R

Alternatively, you can load a specific version of R using one of the following

module load apps/R/3.4.0/gcc-4.8.5module load apps/R/3.3.3/gcc-4.8.5module load apps/R/3.3.2/gcc-4.8.5

R can then be run with

$ R


Here, we assume that you wish to run the program my_code.r on the system. With batch usage it is recommendedto load a specific version of R, for example module load apps/R/3.4.0, to ensure the expected output isachieved.

2.3. ShARC 243




First, you need to write a batch submission file. We assume you’ll call this my_job.sge

#!/bin/bash#$ -cwd # Run job from current directory#$ -l rmem=4G # Request 4 gigabytes of memory

module load apps/R/3.4.0/gcc-4.8.5 # Recommended to load a specific version of R

R CMD BATCH my_code.r my_code.r.o$JOB_ID

Note that R must be called with both the CMD and BATCH options which tell it to run an R program, in this casemy_code.r. If you do not do this, R will attempt to open an interactive prompt.

The final argument, my_code.r.o$JOBID, tells R to send output to a file with this name. Since $JOBID willalways be unique, this ensures that all of your output files are unique. Without this argument R sends all output to afile called my_code.Rout.

Ensuring that my_code.r and my_job.sge are both in your current working directory, submit your job to thebatch system

qsub my_job.sge

Replace my_job.sge with the name of your submission script.

Warning: By default, R will save variables in the workspace in the current directory (in a file called .RData)when a script exits and reload them when it starts from the same directory. To disable this, add the –no-save and–no-restore options to your command, e.g. R CMD BATCH –no-save –no-restore my_code.r my_code.r.o$JOB_ID.

Graphical output

By default, graphical output from batch jobs is sent to a file called Rplots.pdf


As you will not have permissions to install packages to the default folder, additional R packages can be installed toyour home folder ~/. To create the appropriate folder, install your first package in R in interactive mode. Load aninteractive R session as described above, and install a package with

install.packages()

You will be prompted to create a personal package library. Choose yes. The package will download and install from aCRAN mirror (you may be asked to select a nearby mirror, which you can do simply by entering the number of yourpreferred mirror).

Once the chosen package has been installed, additional packages can be installed either in the same way, or by creatinga .R script. An example script might look like

install.packages("dplyr")install.packages("devtools")

Call this using source(). For example if your script is called packages.R and is stored in your home folder, source thisfrom an interactive R session with



source("~/packages.R")

These additional packages will be installed without prompting to your personal package library.

To check your packages are up to date, and update them if necessary, run the following line from an R interactivesession

update.packages(lib.loc = "~/R/x86_64-unknown-linux-gnu-library/3.4/")

The folder name after ~/R/ will likely change, but this can be completed with tab autocompletion from the R session.Ensure lib.loc folder is specified, or R will attempt to update the wrong library.

Warning: Notice that the personal package library path includes the version of R: if after installing some packagesyou switch to using a different major or minor version of R then you will need then to install those package for thisnew version.

R Packages that require external libraries

Some R packages require external libraries to be installed before you can install and use them. Since there are somany, we only install those libraries that have been explicitly requested by users of the system.

The associated R packages are not included in the system install of R, so you will need to install them yourself to yourhome directory following the instructions linked to below.

• geos This is the library required for the rgeos package.

• Geospatial Data Abstraction Library (GDAL) and PROJ.4 These are the libraries required for the rgdal pack-age.

Warning: To install r packages that require external libraries, the libraries need to be loaded prior to installingthe r packages. E.g. to install package rgeos you would need to load geos, enter an interactive R session and theninstall rgeos

module load libs/geos/3.6.1/gcc-4.9.4Rinstall.packages("rgeos")

See here more information on the available external libraries

Using the Rmath library in C Programs

The Rmath library allows you to access some of R’s functionality from a C program. For example, consider theC-program below

#include <stdio.h>#define MATHLIB_STANDALONE#include "Rmath.h"

main(){double shape1,shape2,prob;

shape1 = 1.0;shape2 = 2.0;

2.3. ShARC 245

http://semver.org/


prob = 0.5;

printf("Critical value is %lf\n",qbeta(prob,shape1,shape2,1,0));}

This makes use of R’s qbeta function. You can compile and run this on a worker node as follows.

Start a session on a worker node with qrsh or qrshx and load the R module

module load apps/R/3.4.0/gcc-4.8.5

Assuming the program is called test_rmath.c, compile with

gcc test_rmath.c -lRmath -lm -o test_rmath

For full details about the functions made available by the Rmath library, see section 6.7 of the document Writing Rextensions

Versions of R with faster linear algebra

We have compiled versions of R using the Intel Compiler suite and the Intel MKL. These can be faster than this‘standard’ version in some cases. For more details see R (Intel Build)

Installation Notes


version 3.4.0


This was a scripted install. It was compiled from source with gcc 4.8.5 and with --enable-R-shlib enabled. Itwas run in batch mode.

• install_r_3.4.0_gcc4.8.5.sh Downloads, compiles, tests and installs R 3.4.0 and the Rmath library.

• R 3.4.0 Modulefile located on the system at /usr/local/modulefiles/apps/R/3.4.0/

• Install log-files, including the output of the make check tests are available on the system at/usr/local/packages/apps/R/3.4.0/gcc-4.8.5/install_logs/

version 3.3.3






version 3.3.2






https://stat.ethz.ch/pipermail/r-help//2017-March/445277.html







SAMtools

SAMtools

Version 1.7

Dependancies gcc/4.9.4

URL https://github.com/samtools/samtools/releases/download/1.7/samtools-1.7.tar.bz2

Documentation http://www.htslib.org/doc/samtools.html

SAM (Sequence Alignment/Map) format is a generic format for storing large nucleotide sequence alignments. SAMaims to be a format that is

• flexible enough to store all the alignment information generated by various alignment programs

• simple enough to be easily generated by alignment programs or converted from existing alignment formats

• compact in file size

• allows most of operations on the alignment to work on a stream without loading the whole alignment intomemory

• allows the file to be indexed by genomic position to efficiently retrieve all reads aligning to a locus

SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging,indexing and generating alignments in a per-position format.

Usage

SAMtools can be activated using the module file:

module load apps/SAMtools/1.7/gcc-4.9.4

Note: The module file also loads the compiler gcc/4.9.4.

Test

1. Download the following example files using:

cd some_directorycp -r /usr/local/packages/apps/SAMtools/1.7/gcc-4.9.4/examples .cd examples# sample data# ex1.fa - contains 2 sequences cut from the human genome

2.3. ShARC 247

https://github.com/samtools/samtools/releases/download/1.7/samtools-1.7.tar.bz2

http://www.htslib.org/doc/samtools.html


# ex1.sam.gz - contains MAQ alignments# 00README.txt contains instructions and description of resultssamtools #gives a list of optionssamtools faidx ex1.fa #index the reference FASTAsamtools view -S -b -t ex1.fa.fai -o ex1.bam ex1.sam.gz #convert SAM to BAMsamtools index ex1.bam #build index for BAMsamtools view ex1.bam seq2:450-550 #view a portion of BAM filesamtools tview -p seq2:450 ex1.bam exl.fa #visually inspect alignments at same→˓locationsamtools mpileup -f ex1.fa ex1.bam #view data in pileup formatsamtools mpileup -vu -f ex1.fa ex1.bam > ex1.vcf #generate uncompressed VCF file→˓of variantssamtools mpileup -g -f ex1.fa ex1.bam > ex1.bcf #generate compressed VCF file of→˓variants

Installation notes

SAMtools was compiled using the install_SAMtools.sh script, the module file is /usr/local/modulefiles/apps/SAMtools/1.7/gcc-4.9.4.

Abaqus

Abaqus

Versions 2017, 6.14-2

Dependencies Intel compiler 15.0.7 and Foxit modules loaded. User subroutines need Intel compiler2011 or above, GCC 4.1.2 or above.

URL http://www.3ds.com/products-services/simulia/products/abaqus/

Documentation https://www.3ds.com/support/documentation/users-guides/

Local URL https://www.shef.ac.uk/wrgrid/software/abaqus

Abaqus is a software suite for Finite Element Analysis (FEA) developed by Dassault Systèmes.

Usage

Abaqus version 2017 or version 6.14-2 can be activated using the module files:

module load apps/abaqus/2017/binarymodule load apps/abaqus/6.14-2/binary

Type abaqus cae to launch the Abaqus GUI from an interactive session with X Window support (e.g. an interactiveqsh session). Type abaqus for the command line interface. Typing abaqus -help will display a list of usageoptions.

The PDF viewer foxit can be launched to view the PDF documentation for Abaqus 6.14-2 located at /usr/local/packages/apps/abaqus/6.14-2/binary/Documentation/docs/v6.14/pdf_books.

Abaqus 2017 documentation in HTML format is located at /usr/local/packages/apps/abaqus/2017/binary/Documentation/DSSIMULIA_Established_homepage_English.htm.


http://www.3ds.com/products-services/simulia/products/abaqus/

https://www.3ds.com/support/documentation/users-guides/

https://www.shef.ac.uk/wrgrid/software/abaqus


Note: When using hardware-accelerated graphics rendering for Abaqus 6.14-2 on Sharc, i.e., during a qsh-visinteractive session, please run abq6142 cae to launch the GUI. When using a general compute node for Abaqus2017 on Sharc, please run abaqus cae -mesa or abq2017 cae -mesa to launch the GUI without support forhardware-accelerated graphics rendering. The option -mesa disables hardware-accelerated graphics rendering withinAbaqus’s GUI.

Abaqus example problems

Abaqus contains a large number of example problems which can be used to become familiar with Abaqus on thesystem. These example problems are described in the Abaqus documentation and can be obtained using the Abaqusfetch command. For example, after loading the Abaqus module enter the following at the command line to extractthe input file for test problem s4d:

abaqus fetch job=s4d

This will extract the input file s4d.inp to run the computation defined by the commands and batch submission scriptbelow.

Batch jobs

The easiest way of running a batch job for a particular version of Abaqus (e.g. 6.14-2) is:

module load apps/abaqus/6.14-2/binaryrunabaqus

The runabaqus command submits an Abaqus input file into the batch queuing system and can take a number ofdifferent parameters according to your requirements. Typing runabaqus will display information about its usage.Note: runabaqus is not setup for use with Abaqus 2017.

Users are encouraged to write their own batch submission scripts. The following is an example batch submission script,my_job.sh, to run the executable abq6142 and which is submitted to the queue by typing qsub my_job.sh:

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00#$ -l rmem=2G#$ -pe smp 4

module load apps/abaqus/6.14-2/binary

abq6142 job=my_job input=s4d.inp scratch=$TMPDIR memory="2gb" interactive mp_→˓mode=threads cpus=$NSLOTS

The above script requests 4 cores using the OpenMP parallel environment smp with a runtime of 30 mins and 2 GBof real memory per core. The Abaqus input file is s4d.inp.

User subroutines: The script below is an example of a batch submission script for a single core job with a runtime of30 mins, 8 GB of real memory and with user subroutine umatmst3.f and input file umatmst3.inp:

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00#$ -l rmem=8G

module load apps/abaqus/6.14-2/binary

2.3. ShARC 249


abq6142 job=my_user_job input=umatmst3.inp user=umatmst3.f scratch=$TMPDIR memory="8gb→˓" interactive

The input file umatmst3.inp and the Fortran user subroutine umatmst3.f are obtained by typing abaqusfetch job=umatmst3*. Note that the module dev/intel-compilers/15.0.7, required for user subrou-tines, is automatically loaded when the module for Abaqus is loaded.

Important information: Please note that at present Abaqus will not run on more than one node when using MPI onShARC. The SGE option -l excl=true can be used to request that an MPI job runs on one compute node only.

Licensed options

All available Abaqus licenses can be viewed using abaqus licensing r e.g.

$ module load apps/abaqus/2017/binary$ abaqus licensing r

Feature Version #licenses Expires Vendor_______ _________ _________ __________ ______abaqus_extended 61.9 19 31-dec-2018 ABAQUSLMabaqus 61.9 250 31-dec-2018 ABAQUSLMams 61.9 1 31-dec-2018 ABAQUSLMaqua 61.9 250 31-dec-2018 ABAQUSLMcosim_acusolve 61.9 1 31-dec-2018 ABAQUSLMcosim_direct 61.9 1 31-dec-2018 ABAQUSLMcse 61.9 1 31-dec-2018 ABAQUSLMdesign 61.9 250 31-dec-2018 ABAQUSLMeuler_lagrange 61.9 1 31-dec-2018 ABAQUSLMgpgpu 61.9 1 31-dec-2018 ABAQUSLMmultiphysics 61.9 1 31-dec-2018 ABAQUSLMparallel 61.9 16384 31-dec-2018 ABAQUSLMsw_assoc_import 61.9 1 31-dec-2018 ABAQUSLMcatiav5_assoc_import 61.9 1 31-dec-2018 ABAQUSLMcatiav5_import 61.9 1 31-dec-2018 ABAQUSLMcatiav6_assoc_import 61.9 1 31-dec-2018 ABAQUSLMtomee 61.9 1 31-dec-2018 ABAQUSLMpydriver 61.9 1 31-dec-2018 ABAQUSLMcae 61.9 19 31-dec-2018 ABAQUSLMrtgateway 61.9 19 31-dec-2018 ABAQUSLMgateway 61.9 19 31-dec-2018 ABAQUSLMsafe_ex_gui 61.9 19 31-dec-2018 ABAQUSLMcfd 61.9 250 31-dec-2018 ABAQUSLMexplicit 61.9 250 31-dec-2018 ABAQUSLMfoundation 61.9 250 31-dec-2018 ABAQUSLMsimflow 61.9 250 31-dec-2018 ABAQUSLMstandard 61.9 250 31-dec-2018 ABAQUSLMcse_token 61.9 250 31-dec-2018 ABAQUSLMsafe_ex_engine 61.9 250 31-dec-2018 ABAQUSLMtosca_topo 61.9 250 31-dec-2018 ABAQUSLMtosca_shape 61.9 250 31-dec-2018 ABAQUSLMtosca_bead 61.9 250 31-dec-2018 ABAQUSLMtosca_sizing 61.9 250 31-dec-2018 ABAQUSLMtosca_int_abaqus 61.9 250 31-dec-2018 ABAQUSLMtosca_int_ansys 61.9 250 31-dec-2018 ABAQUSLMtosca_int_nastran 61.9 250 31-dec-2018 ABAQUSLMtosca_adv_nonlinear 61.9 250 31-dec-2018 ABAQUSLM



tosca_adv_durability 61.9 250 31-dec-2018 ABAQUSLMtosca_adv_morph 61.9 250 31-dec-2018 ABAQUSLMtosca_smooth 61.9 250 31-dec-2018 ABAQUSLMtosca_report 61.9 250 31-dec-2018 ABAQUSLMtfluid_topo 61.9 250 31-dec-2018 ABAQUSLMtfluid_smooth 61.9 250 31-dec-2018 ABAQUSLMtfluid_parallel 61.9 250 31-dec-2018 ABAQUSLMtfluid_int_ccmp 61.9 250 31-dec-2018 ABAQUSLMtfluid_int_fluent 61.9 250 31-dec-2018 ABAQUSLM

Run abaqus licensing for usage info for the Abaqus licensing sub-command.

Installation notes

Abaqus 2017 was installed using the install_abaqus_2017.sh script; the module file is /usr/local/modulefiles/apps/abaqus/2017/binary.

Abaqus 6.14-2 was installed using the install_abaqus.sh script; the module file is /usr/local/modulefiles/apps/abaqus/6.14-2/binary.

The binary installations were tested by launching abaqus cae and by using the above batch submission scripts.Abaqus at present does not run on more than one node when using MPI due to password-less ssh being disabled acrossnodes on ShARC.

ANSYS

ANSYS

Versions 16.1, 17.2, 18.0, 18.2

Dependencies No prerequsite modules loaded. However, If using the User Defined Functions (UDF)will also need the following: For ANSYS Mechanical, Workbench, CFX and AutoDYN: Intel14.0 or above; Compiler For Fluent: GCC 4.6.1 or above

URL http://www.ansys.com


The Ansys suite of programs can be used to numerically simulate a large variety of structural and fluid dynamicsproblems found in many engineering, physics, medical, aeronotics and automative industry applications.

Usage

Ansys can be activated using the module files:

module load apps/ansys/16.1module load apps/ansys/17.2module load apps/ansys/18.0/binarymodule load apps/ansys/18.2/binary

The Ansys Workbench GUI executable is ansyswb. ansyswb can be launched during an interactive session with XWindow support (e.g. an interactive qsh session). The Ansys exectuable is ansys and fluent is the executable forFluent. Typing ansys -g or fluent -g -help will display a list of usage options.

2.3. ShARC 251

http://www.ansys.com



Batch jobs

The easiest way of running batch jobs for a particular version of Ansys (e.g. 17.2) is:

module load apps/ansys/17.2runansys

Or the same for Fluent:

module load apps/ansys/17.2runfluent

The runfluent and runansys commands submit a Fluent journal or Ansys input file into the batch system andcan take a number of different parameters, according to your requirements. Typing runansys or runfluent willdisplay information about their usage. Note: runansys and runfluent are not setup for use with Ansys 18.0 and18.2.

Users are encouraged to write their own batch submission scripts. The following is an example batch submission script,my_job.sh, to run fluent version 16.1 and which is submitted to the queue by typing qsub my_job.sh:

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00#$ -l rmem=2G#$ -pe mpi-rsh 8

module load apps/ansys/16.1

fluent 2d -i flnt.inp -g -t8 -sge -mpi=intel -rsh -sgepe mpi-rsh

The script requests 8 cores using the MPI parallel environment mpi-rsh with a runtime of 30 mins and 2 GB of realmemory per core. The Fluent input file is flnt.inp. Note: Please use the mpi-rsh parallel environment to runMPI parallel jobs for Ansys/Fluent.

Installation notes

Ansys 16.1 was installed using the install_ansys.sh script; the module file is /usr/local/modulefiles/apps/ansys/16.1.

Ansys 17.2 was installed using the install_ansys.sh script; the module file is /usr/local/modulefiles/apps/ansys/17.2.

Ansys 18.0 was installed using the install_ansys_180.sh script; the module file is /usr/local/modulefiles/apps/ansys/18.0/binary.

Ansys 18.2 was installed using the install_ansys_182.sh script; the module file is /usr/local/modulefiles/apps/ansys/18.2/binary.

The binary installations were tested by launching ansyswb and by using the above batch submission script. Thempi-rsh tight-integration parallel environment is required to run Ansys/Fluent using MPI due to password-less sshbeing disabled across nodes on ShARC.

Ansys EM



Ansys EM

Versions 16.1, 17.2, 18.0, 18.2

Dependencies For integration with Ansys Workbench requires Ansys 16.1, 17.2, 18.0, 18.2

URL http://www.ansys.com


ANSYS Electromagnetics Suite for Linux/Unix.

Usage

Ansys EM can be activated using the module files:

module load apps/ansysem/16.1module load apps/ansysem/17.2module load apps/ansysem/18.0/binarymodule load apps/ansysem/18.2/binary

Ansys EM is integrated with the Ansys Workbench GUI (the ansyswb executable) for each version. The Ansys EMexectuable is ansysedt.

Note: An accelerated-graphics interactive session with X Window support (i.e. a Hardware-accelerated graphicsrendering interactive session on ShARC) is required to run the ansysedt executable as a GUI. On the login node,the command qsh-vis will initiate an accelerated-graphics interactive session.

Batch jobs

The following is an example batch submission script which is submitted to the queue by typing qsub my_job.sh:

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00#$ -l rmem=2G#$ -pe mpi 8

module load apps/ansysem/16.1

ansysedt -ng -BatchSolve -Distributed -machinelist num=8 -batchoptions 'HFSS/→˓HPCLicenseType'='pool' -batchoptions 'HFSS-IE/HPCLicenseType'='pool' Tee.aedt

The script requests 8 cores using the MPI parallel environment mpi with a runtime of 30 mins and 2 GB of realmemory per core. The Ansys EM input file is Tee.aedt and batch options 'HFSS/HPCLicenseType'='pool'and 'HFSS-IE/HPCLicenseType'='pool' change the HPC licencing from “pack” (the default) to “pool”.

Installation notes

Ansys EM 16.1 was installed using the install_ansysem.sh script; the module file is /usr/local/modulefiles/apps/ansysem/16.1.

Ansys EM 17.2 was installed using the install_ansysem.sh script; the module file is /usr/local/modulefiles/apps/ansysem/17.2.

2.3. ShARC 253

http://www.ansys.com



Ansys EM 18.0 was installed using the install_ansysem_180.sh script; the module file is /usr/local/modulefiles/apps/ansysem/18.0/binary.

Ansys EM 18.2 was installed using the install_ansysem_182.sh script; the module file is /usr/local/modulefiles/apps/ansysem/18.2/binary.

The binary installations were tested using ansyswb and the above batch submission script.

BWA

BWA

Version 0.7.17

Dependency Module loaded for GCC compiler 6.2.0

URL http://bio-bwa.sourceforge.net

Documentation http://bio-bwa.sourceforge.net/bwa.shtml

BWA is a software package for mapping low-divergent sequences against a large reference genome, such as the humangenome. It consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM. The first algorithm is designedfor Illumina sequence reads up to 100bp, while the rest two for longer sequences ranged from 70bp to 1Mbp. BWA-MEM and BWA-SW share similar features such as long-read support and split alignment, but BWA-MEM, which isthe latest, is generally recommended for high-quality queries as it is faster and more accurate. BWA-MEM also hasbetter performance than BWA-backtrack for 70-100bp Illumina reads.

Usage

BWA 0.7.17 can be activated using the module file:

module load apps/bwa/0.7.17/gcc-6.2

The BWA executable is bwa. Type bwa for a list of commands.

Installation notes

BWA 0.7.17 was installed using the install_bwa.sh script; the module file is /usr/local/modulefiles/apps/bwa/0.7.17/gcc-6.2. The installation of BWA 0.7.17 was compiled with GCC 6.2.0.

Caffe

Caffe

URL http://caffe.berkeleyvision.org/

Caffe is a Deep Learning framework made with expression, speed, and modularity in mind. It is developed by theBerkeley Vision and Learning Center (BVLC) and by community contributors.


http://bio-bwa.sourceforge.net

http://bio-bwa.sourceforge.net/bwa.shtml

http://caffe.berkeleyvision.org/

http://caffe.berkeleyvision.org/

http://bvlc.eecs.berkeley.edu/


About Caffe on ShARC

A GPU-enabled worker node must be requested in order to use the GPU version of this software. See UsingGPUs on ShARC for more information.

Caffe is available on ShARC as both Singularity images and as a module.

This software and documentation is maintained by the RSES group and GPUComputing@Sheffield_. For featurerequests or if you encounter any problems, please raise an issue on the GPU Computing repository.

Caffe Singularity Images

Singularity images are self-contained virtual machines similar to Docker. For more information on Singularity andhow to use the images, see Singularity.

A symlinked file is provided that always point to the latest image:

#CPU Caffe/usr/local/packages/singularity/images/caffe/cpu.img

#GPU Caffe/usr/local/packages/singularity/images/caffe/gpu.img

To get a bash terminal in to an image for example, use the command:

singularity exec --nv /usr/local/packages/singularity/images/caffe/gpu.img /bin/bash

The exec command can also be used to call any command/script inside the image e.g.

singularity exec --nv /usr/local/packages/singularity/images/caffe/gpu.img caffe→˓train -solver=your_solver.prototxt

The --nv flag enables the use of GPUs within the image and can be removed if the software you’re using doesnot use the GPU.

You may get a warning similar to groups: cannot find name for group ID ..., this can be ignoredand will not have an affect on running the image.

The paths /fastdata, /data, /home, /scratch, /shared are automatically mounted to your ShARC filestoredirectories. For GPU-enabled images the /nvlib and /nvbin is mounted to the correct Nvidia driver version forthe node that you’re using.

To submit jobs that uses a Singularity image, see Submitting Batch Jobs That Uses Singularity Images for moredetail.

Image Index

Paths to the actual images and definition files are provided below for downloading and building of custom images.

• Shortcut to Latest Image

– CPU

* /usr/local/packages/singularity/images/caffe/cpu.img

– GPU

* /usr/local/packages/singularity/images/caffe/gpu.img

2.3. ShARC 255


mailto:GPUComputing@Sheffield_

https://github.com/RSE-Sheffield/GPUComputing/issues


• CPU Images

– Latest: 1.0.0-CPU-Ubuntu16.04 (Python 2.7)

* Path: /usr/local/packages/singularity/images/caffe/1.0.0-cpu-ubuntu16.04.img

– rc3-CPU-Ubuntu16.04 (Python 2.7)

* Path: /usr/local/packages/singularity/images/caffe/rc3-CPU-Ubuntu16.04.img

– Def file: /sharc/software/apps/singularity/caffe_cpu.def

• GPU Images

– Latest: 1.0.0-GPU-Ubuntu16.04-CUDA8-cudNN5.0 (Python 2.7)

* Path: /usr/local/packages/singularity/images/caffe/1.0.0-gpu-ubuntu16.04-cuda8-cudnn6.0.img

– rc3-GPU-Ubuntu16.04-CUDA8-cudNN5.0 (Python 2.7)

* Path: /usr/local/packages/singularity/images/caffe/rc3-GPU-Ubuntu16.04-CUDA8-cudNN5.0.img

– Def file: /sharc/software/apps/singularity/caffe_gpu.def

Using the Caffe Module

First request a GPU interactive session (see Interactive use of the GPUs).

The Caffe module can be loaded with the following command:

module load apps/caffe/rc5/gcc-4.9.4-cuda-8.0-cudnn-5.1

Installing Additional Python Modules (Optional)

The Caffe module is pre-installed with Anaconda version 3.4.2. You can install additional python packages by creatinga virtual python environment in your home directory using conda.

#Creates a conda environment named caffeconda create -n caffe python=3.5

#Activates the caffe python environmentsource activate caffe

You will also need to install numpy which can be obtained from the conda repository.

conda install numpy


If you created a virtual python environment, you must activate it at every new session and within your job scripts:


#Activation below is only needed if you've installed your on python modulessource activate caffe



Caffe Training

GPUComputing@Sheffield_ provides training materials on the use of Caffe on the DGX-1 and ShARC cluster.

Installation Notes

For the module:


The following modules are automatically loaded:

• GCC 4.9.4

• CUDA 8

• cuDNN 5.1

And comes with the following libraries:

• Anaconda 4.2.0 (Python 3)

• boost

• protobuf

• hdf5

• snappy

• glog

• gflags

• openblas

• leveldb

• lmdb

• yasm

• libx264

• libx265

• libfdk_acc

• libopus

• libogg

• libvorbis

• freetype

• ffmpeg

• libjpeg

• libpng

• libtiff

2.3. ShARC 257

mailto:GPUComputing@Sheffield_

http://gpucomputing.shef.ac.uk/education/intro_dl_sharc_dgx1/


• opencv 3.2.0

Code Saturne

Code Saturne

Versions 4.0.7, 5.0.4

Dependencies GCC compiler, Open MPI, Scotch/PT-Scotch and Libxml2. Modules loaded for GCC4.9.4 and Open MPI 1.10.4 or GCC 6.2.0 and Open MPI 2.0.1 for version 4.0.7; Intel 17.0.0 andOpen MPI 2.0.1 for version 5.0.4

URL http://code-saturne.org/cms/

Documentation http://code-saturne.org/cms/documentation

Code_Saturne solves the Navier-Stokes equations for 2D, 2D-axisymmetric and 3D flows, steady or unsteady, laminaror turbulent, incompressible or weakly dilatable, isothermal or not, with scalars transport if required.

Usage

Code Saturne 4.0.7 or 5.0.4 can be activated using the module files:

module load apps/code_saturne/4.0.7/gcc-4.9.4-openmpi-1.10.4module load apps/code_saturne/4.0.7/gcc-6.2-openmpi-2.0.1module load apps/code_saturne/5.0.4/intel-17.0.0-openmpi-2.0.1

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submission script,run_solver.sh, to run Code Saturne 4.0.7 and which is submitted to the queue by typing qsub run_solver.sh.:


# Load environment if this script is run directly.module load apps/code_saturne/4.0.7/gcc-4.9.4-openmpi-1.10.4

# Run solver.mpiexec -n 4 cs_solver --param case1.xml --mpi $@export CS_RET=$?

exit $CS_RET

The script requests 4 cores using the MPI parallel environment mpi with a runtime of 30 mins and 2G of real mem-ory per core. The Code Saturne input file is case1.xml. Note: The above batch script was adapted from therun_solver script generated during the batch usage test (see Installation notes below).


http://code-saturne.org/cms/

http://code-saturne.org/cms/documentation


Installation notes

Code Saturne 4.0.7 was installed with GCC 4.9.4 and Open MPI 1.10.4 using the install_code_saturne.sh script; the module file is /usr/local/modulefiles/apps/code_saturne/4.0.7/gcc-4.9.4-openmpi-1.10.4.

Code Saturne 4.0.7 was installed with GCC 6.2.0 and Open MPI 2.0.1 using the install_code_saturne.sh script; the module file is /usr/local/modulefiles/apps/code_saturne/4.0.7/gcc-6.2-openmpi-2.0.1.

Code Saturne 5.0.4 was installed with Intel 17.0.0 and Open MPI 2.0.1 using the install_code_saturne_5.0.4.sh script; the module file is /usr/local/modulefiles/apps/code_saturne/5.0.4/intel-17.0.0-openmpi-2.0.1.

The above installations used the libraries Scotch/PT-Scotch 6.0.4 and Libxml2 2.9.1. No additional, optional librarieswere used during compilation.

Post-installation: Please read the instructions at the end of the install script for the Code Saturne files to manuallyedit.

The installation of Code Saturne 4.0.7 was tested using the following example calculations.

Interactive usage test:

$ module load apps/code_saturne/4.0.7/gcc-4.9.4-openmpi-1.10.4$ code_saturne create -s T_junction -c case1$ cp $build_dir/examples/1-simple_junction/case1/case1.xml ./T_junction/case1/DATA$ cp $build_dir/examples/1-simple_junction/mesh/downcomer.des ./T_junction/MESH$ cd ./T_junction/case1/DATA$ code_saturne run --param case1.xml$ cd ../RESU/yyyymmdd-hhmm

The output ./T_junction/case1/RESU/yyyymmdd-hhmm/listing file should contain “END OF CAL-CULATION”.

Batch usage test using the parallel environment mpi 4 (performed after the interactive test above):

$ cd ./T_junction/case1/DATA$ code_saturne run --initialize --param case1.xml --nprocs 4$ cd ../RESU/yyyymmdd-hhmm$ vi run_solver$ qsub run_solver

The output ./T_junction/case1/RESU/yyyymmdd-hhmm/listing file should contain “END OF CAL-CULATION”.

User subroutines test (performed after the above two tests):

$ cd ./T_junction/case1/SRC$ cp ./REFERENCE/cs_user_parameters.f90 .$ code_saturne compile

DL_POLY

DL_POLY

2.3. ShARC 259


Version 4.08

Dependencies GCC compiler and Open MPI. Modules loaded for GCC 6.2.0, Open MPI 2.1.1 andPLUMED 2.3.2.

URL https://www.scd.stfc.ac.uk/Pages/DL_POLY.aspx

Documentation ftp://ftp.dl.ac.uk/ccp5/DL_POLY/DL_POLY_4.0/DOCUMENTS/USRMAN4.pdf

DL_POLY is a general purpose classical molecular dynamics (MD) simulation software developed at Daresbury Lab-oratory by I.T. Todorov and W. Smith.

Usage

DL_POLY 4.08 can be activated using the module file:

module load apps/dl_poly/4.08/gcc-6.2-openmpi-2.1.1

The DL_POLY executable is DLPOLY.Z. Three DL_POLY input files CONFIG, CONTROL and FIELD are requiredin the directory where you run your job.

Batch usage

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run DLPOLY.Z and which is submitted to the queue by typing qsub my_job.sh.


module load apps/dl_poly/4.08/gcc-6.2-openmpi-2.1.1

mpirun DLPOLY.Z

The script requests 8 cores using the MPI parallel environment mpi with a runtime of 30 mins and 2 GB of realmemory per core. Note: If your OUTPUT file appears truncated when running a DL_POLY job using MPI, then addthe l_scr keyword to your CONTROL file to place the output in the standard output file (see section 5.2.5 ParallelI/O in the user manual).

Installation notes

DL_POLY 4.08 was installed using the install_dl_poly.sh script; the module file is gcc-6.2-openmpi-2.1.1.

During compilation DL_POLY 4.08 was optionally interfaced with the PLUMED 2.3.2 library.

The installation of DL_POLY 4.08 was tested using TEST 28 (see the above installation script and ftp://ftp.dl.ac.uk/ccp5/DL_POLY/DL_POLY_4.0/DATA/ for details).


https://www.scd.stfc.ac.uk/Pages/DL_POLY.aspx

ftp://ftp.dl.ac.uk/ccp5/DL_POLY/DL_POLY_4.0/DOCUMENTS/USRMAN4.pdf

ftp://ftp.dl.ac.uk/ccp5/DL_POLY/DL_POLY_4.0/DATA/

ftp://ftp.dl.ac.uk/ccp5/DL_POLY/DL_POLY_4.0/DATA/


DL_POLY_CLASSIC

DL_POLY_CLASSIC

Version 1.9

Dependencies GCC compiler and Open MPI. Modules loaded for GCC 6.2.0, Open MPI 2.1.1

URL https://ccpforge.cse.rl.ac.uk/gf/project/dl_poly_classic/

Documentation https://www.ccp5.ac.uk/sites/www.ccp5.ac.uk/files/dl_poly_classic/USRMAN.pdf

DL_POLY_CLASSIC is a general purpose classical molecular dynamics (MD) simulation software developed atDaresbury Laboratory by I.T. Todorov and W. Smith.

Usage

DL_POLY_CLASSIC 1.9 can be activated using the module file:

module load apps/dl_poly_classic/1.9/gcc-6.2-openmpi-2.1.1

The DL_POLY_CLASSIC executable is DLPOLY.X. Three DL_POLY input files CONFIG, CONTROL and FIELDare required in the directory where you run your job.

Batch usage

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run DLPOLY.X and which is submitted to the queue by typing qsub my_job.sh.


module load apps/dl_poly_classic/1.9/gcc-6.2-openmpi-2.1.1

mpirun DLPOLY.X

The script requests 8 cores using the MPI parallel environment mpi with a runtime of 30 mins and 2 GB of realmemory per core. Note: If your OUTPUT file appears truncated when running a DL_POLY_CLASSIC job using MPI,then add the l_scr keyword to your CONTROL file to place the output in the standard output file.

Installation notes

DL_POLY_CLASSIC 1.9 can be installed using the install_dl_poly_classic.sh script; the module file isgcc-6.2-openmpi-2.1.1.

The installation of DL_POLY_CLASSIC was tested using TEST01 for DL_POLY_CLASSIC (https://ccpforge.cse.rl.ac.uk/gf/download/frsrelease/145/8489/TEST1.tar.gz)

2.3. ShARC 261

https://ccpforge.cse.rl.ac.uk/gf/project/dl_poly_classic/

https://www.ccp5.ac.uk/sites/www.ccp5.ac.uk/files/dl_poly_classic/USRMAN.pdf

https://ccpforge.cse.rl.ac.uk/gf/download/frsrelease/145/8489/TEST1.tar.gz

https://ccpforge.cse.rl.ac.uk/gf/download/frsrelease/145/8489/TEST1.tar.gz


Foxit

Foxit

Version 2.4.0

Dependancies

•

URL https://www.foxitsoftware.com

Documentation https://help.foxitsoftware.com

Foxit Reader provides a full set of feature rich tools to help you comment on PDF files easily. Comment tools includethe Text Markup tools, Typewriter, Note tool, Drawing tools, and stamp tools. You can also click the pin icon in theComments toolbar to keep the tool selected, so you can use the tools continuously without having to select the specifictool after each use.

Usage

Foxit can be activated using the module file:

module load apps/foxit

Installation notes

Foxit was installed using the install_foxit.sh script, the module file is 2.4.0.

Gaussian 09

Gaussian 09

Version D.01

Dependencies Portland Group Fortran compiler and GCC compiler. Module for PGI 17.5 loaded.

URL http://gaussian.com

Documentation http://gaussian.com/techsupport/

Gaussian 09 provides state-of-the-art capabilities for electronic structure modelling. GaussView is the graphical inter-face used with Gaussian.

Usage

Gaussian 09 Revision D.01 and GaussView 5.0.8, the GUI for Gaussian 09, can be activated using the module file:

module load apps/gaussian_09/d.01/pgi-17.5


https://www.foxitsoftware.com

https://help.foxitsoftware.com

http://gaussian.com

http://gaussian.com/techsupport/


The GaussView executable is gv. gv can be launched from an interactive session with X Window support (e.g. aninteractive qsh session). The Gaussian 09 command line executuable is g09.

Important note: Access to Gaussian 09 on ShARC is available to all users from the Department of Chemistryby default (i.e. user IDs beginning with ch); other users wishing to access Gaussian 09 will need to [email protected] to request access (i.e. to be added to unix group gaussian09).

Gaussian 09 utilities

The Gaussian 09 utilities can be accessed by loading the Gaussian 09 module file (see above). The utilities are executedby typing their name, e.g., formchk and cubegen. The full list of utilities is described at http://gaussian.com/utils.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run g09 and which is submitted to the queue by typing qsub my_job.sh.


module load apps/gaussian_09/d.01/pgi-17.5

export GAUSS_SCRDIR=$TMPDIR

g09 my_input.gjf

The script requests 2 cores using the shared memory parallel environment smp with a runtime of 30 mins and 2GB ofreal memory per core. The Gaussian 09 input file is my_input.gjf.

Please note: The above script specifies that the scratch directory for Gaussian 09, GAUSS_SCRDIR, is set to$TMPDIR on a compute node.

Installation notes

Gaussian 09 Revision D.01 and GaussView 5.0.8 were installed using the install_gaussian_09.sh script; themodule file is pgi-17.5. The Gaussian 09 code was compiled with PGI 17.5 and GCC 4.8.5.

The Gaussian 09 installation was tested by running a batch job using the following text (including a blank line at theend) in an input file and the my_job.sh batch script, above.

%chk=h2o.chk%nproc=2%mem=2GB#n hf/6-31G(d,p) opt freq

H2O

0 1OH 1 r1H 1 r2 2 a1

2.3. ShARC 263

http://gaussian.com/utils


r1 1.0r2 1.0a1 105.0

Gnuplot

Gnuplot

Version 5.0.6

Dependencies GCC compiler

URL http://www.gnuplot.info/

Documentation http://www.gnuplot.info/documentation.html

Gnuplot is a portable command-line driven graphing utility for Linux, OS/2, MS Windows, OSX, VMS, and manyother platforms. The source code is copyrighted but freely distributed (i.e., you don’t have to pay for it). It wasoriginally created to allow scientists and students to visualize mathematical functions and data interactively, but hasgrown to support many non-interactive uses such as web scripting. It is also used as a plotting engine by third-partyapplications like Octave. Gnuplot has been supported and under active development since 1986.

Usage

Gnuplot 5.0.6 can be activated using the module file:

module load apps/gnuplot/5.0.6/gcc-4.8.5

Installation notes

Gnuplot 5.0.6 was installed using the install_gnuplot.sh script; the module file is gcc-4.8.5. The instal-lation of Gnuplot 5.0.6 was compiled with GCC 4.8.5 and was tested using the make check command as part ofrunning the above installation script.

GROMACS

GROMACS

Version 2016.4

Dependencies Modules loaded for GCC 4.9.4 and, for GPU support, CUDA 8.0.44.

URL http://www.gromacs.org/

Documentation http://manual.gromacs.org/documentation/

Tutorials http://www.gromacs.org/Documentation/Tutorials


http://www.gnuplot.info/

http://www.gnuplot.info/documentation.html

http://www.gromacs.org/

http://manual.gromacs.org/documentation/

http://www.gromacs.org/Documentation/Tutorials


GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motionfor systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins,lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast atcalculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for researchon non-biological systems, e.g. polymers.

Usage

GROMACS 2016.4 can be activated using the module files:

module load apps/gromacs/2016.4/gcc-4.9.4module load apps/gromacs/2016.4/gcc-4.9.4-cuda-8.0

Where the latter module file apps/gromacs/2016.4/gcc-4.9.4-cuda-8.0 is for the installation compiledto run on GPUs. The GROMACS executable is gmx. Typing gmx help commandswill display a list of commandsfor gmx and their function.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run gmx and which is submitted to the queue by typing qsub my_job.sh.

#!/bin/bash#$ -cwd#$ -l h_rt=06:00:00#$ -l rmem=2G#$ -pe smp 1#$ -l gpu=1

module load apps/gromacs/2016.4/gcc-4.9.4-cuda-8.0

gmx grompp -f grompp.mdp -c conf.gro -p topol.top -o topol.tprgmx mdrun -s topol.tpr

The script requests one CPU core using the OpenMP parallel environment smp, with 2 GB of real memory per CPUcore, and one GPU per one CPU core. The requested runtime is 6 hours. The GROMACS input line beginning withgmx grompp is to make a run input file; gmx mdrun is to perform a simulation, do a normal mode analysis or anenergy minimization.

Installation notes

Two GROMACS 2016.4 installations are available on ShARC; one with and one without GPU support. Both instal-lations use single-node OpenMP parallelism, are single-precision builds and use the GROMACS installation of theFFTW3 library.

GROMACS 2016.4 without GPU support was installed using the install_gromacs.sh script; the module file is/usr/local/modulefiles/apps/gromacs/2016.4/gcc-4.9.4.

GROMACS 2016.4 with GPU support was installed using the install_gromacs_gpu.sh script; the module fileis /usr/local/modulefiles/apps/gromacs/2016.4/gcc-4.9.4-cuda-8.0.

The GROMACS 2016.4 installations were tested by using make check to run regression tests as part of the instal-lation process.

2.3. ShARC 265


GULP

GULP

Latest Version 4.4

URL https://nanochemistry.curtin.edu.au/gulp/

Overview

GULP is a program for performing a variety of types of simulation on materials using boundary conditions of 0-D (molecules and clusters), 1-D (polymers), 2-D (surfaces, slabs and grain boundaries), or 3-D (periodic solids).The focus of the code is on analytical solutions, through the use of lattice dynamics, where possible, rather than onmolecular dynamics. A variety of force fields can be used within GULP spanning the shell model for ionic materials,molecular mechanics for organic systems, the embedded atom model for metals and the reactive REBO potential forhydrocarbons. Analytic derivatives are included up to at least second order for most force fields, and to third order formany.

GULP can either be run:

• on one core (serial build) or

• on multiple cores, potentially on different machines (MPI build).

Licensing

The authors of GULP have given the University of Sheffield permission to install the software on our clusters. Thelicense restrictions are as follows:

• The program is not to be distributed to anyone else without the express permission of the author.

• The program is not to be used for commercial research. For any commercial use of the program a licensemust be obtained from Accelrys Inc, including contract research.

• The program is supplied on an “as is” basis with no implied guarantee or support.

To ensure that the software is not used for commercial purposes access to GULP is restricted to members of the gulp(UNIX) group. To be added to this group, you will need to contact [email protected] and statethe nature of your work.

Interactive Usage

The serial build of GULP should be used for interactive usage. After connecting to ShARC (see Establishing aSSH connection), start an interactive session with the qrsh or qrshx command. Make the serial version of GULPavailable using the following:

module load apps/gulp/4.4/intel-17.0.0

You can then run GULP and show all output on the screen using:

gulp < inputfile

You can also simultaneously save this output in a file (here called outputfile):


https://nanochemistry.curtin.edu.au/gulp/


gulp < inputfile | tee outputfile

If you get:

-bash: gulp: command not found

it is probably because you are not a member of the gulp group. See Licensing.

Serial batch jobs

Create a batch job submission script like the following:

#!/bin/bash#$ -N my_gulp_serial_job#$ -j y#$ -m bea#$ -M [email protected]

module load apps/gulp/4.4/intel-17.0.0

gulp < infile > outfile

Then submit this job to the scheduler:

qsub my_batch_job_script.sge

Multi-node batch jobs

Create a batch job submission script like the following:

#!/bin/bash#$ -N my_gulp_mpi_16_slot_job#$ -pe mpi 16#$ -j y#$ -m bea#$ -M [email protected]

module load apps/gulp/4.4/intel-17.0.0-openmpi-2.0.1

mpirun -np $NSLOTS gulp < infile > outfile

Then submit this job to the scheduler:


Examples

The software comes with several example input files (plus example output files for verifying that the software isworking correctly).

To run them, first activate either serial or the MPI build of GULP (using module load) then copy the examples toa writable location e.g.:

2.3. ShARC 267


cp $GULP_EXAMPLES /data/$USER/gulp_4_4_examplescd /data/$USER/gulp_4_4_examples

Next, create a batch job submission script like the following (for serial testing):

#!/bin/bash#$ -N gulp_ex_serial#$ -j y#$ -m bea#$ -M [email protected]

module purgemodule load apps/gulp/4.4/intel-17.0.0export OMP_NUM_THREADS=1

for infile in ./example*.gin; dooutfile=${infile/gin/got}echo "*******************************************"echo "gulp < $infile | tee $outfile"echo "*******************************************"

gulp < $infile | tee $outfiledone

# Determine the difference between each generated output file# and a corresponding example output file provided with GULPsh ./diff.sh# Collate these differencesfor infile in example*.diff; do

(echo $infile; cat $infile) >> diffs_serial.logdone

or like the following (for MPI testing using 16 cores):

#!/bin/bash#$ -N gulp_ex_mpi_16#$ -pe mpi 16#$ -j y#$ -m bea#$ -M [email protected]

module purgemodule load apps/gulp/4.4/intel-17.0.0-openmpi-2.0.1

for infile in ./example*.gin; dooutfile=${infile/gin/got}echo "*******************************************"echo "mpirun -np 16 gulp < $infile | tee $outfile"echo "*******************************************"

mpirun -np 16 gulp < $infile | tee $outfile

# Needed to avoid errors about not being able to# connect to 'orted' daemons on nodessleep 5

done

# Determine the difference between each generated output file



# and a corresponding example output file provided with GULPsh ./diff.sh# Collate these differencesfor infile in example*.diff; do

(echo $infile; cat $infile) >> diffs_mpi16.logdone

Finally, submit this job to the scheduler:


After receiving email notification that the job has finished, check in the gulp_4_4_examples directory for anoutput file containing:

• the names of the examples that were run;

• timings per example;

• details of any errors

There will also be a diffs*.log file containing details of differences between the generated outputs and the sampleoutputs provided by the software’s authors.

Documentation

See the GULP website or the files in the $GULP_DOCS directory on the cluster.

Installation Notes


Version 4.4

Serial (1 CPU core) and parallel (MPI) builds were compiled. Both builds were compiled with version 17.0.0 of theIntel Fortran compiler and the Intel MKL 2017.1. The MPI build was compiled using OpenMPI 2.0.1.

Both builds were compiled and installed using this script plus

• this serial build configuration

• this MPI build configuration

In addition:

• The serial build modulefile was installed as /usr/local/modulefiles/apps/gulp/4.4/intel-17.0.0

• The parallel build modulefile was installed as /usr/local/modulefiles/apps/gulp/4.4/intel-17.0.0-openmpi-2.0.1

Both versions were tested using the process outlined in the Examples section. The results for the serial version:

• Timings and results file

• Diffs file

A summary of the issues encountered during these 58 tests:

2.3. ShARC 269

https://nanochemistry.curtin.edu.au/gulp/


$ egrep '(WARNING|ERROR)' timings_serial.log | sort | uniq -c1 !! WARNING : Ambiguous vacancy specifier used

The results for the MPI version:

• Timings and results file

• Diffs file

A summary of the issues encountered during these 58 tests:

$ egrep '(WARNING|ERROR)' timings_mpi_16.log | sort | uniq -c1 !! ERROR : RFO keyword cannot be used with conjugate gradients

31 !! ERROR : second derivatives unavailable in parallel1 !! WARNING : Not all configurations optimised successfully in relaxed

IDL

IDL

Version 8.5

Dependencies Java for GUI. Module loaded for Java 1.8.0_102.

URL http://www.exelisvis.co.uk/ProductsServices/IDL.aspx

Documentation http://www.exelisvis.com/docs/using_idl_home.html

IDL is a data analysis language that first appeared in 1977.

Usage

IDL versions can be activated using specific module files:

module load apps/idl/8.5/binarymodule load apps/sswidl/8.5/binary

Then run using idl. Note apps/sswidl/8.5 is the SolarSoft IDL environment.

The IDL licence is restricted - please contact us via helpdesk if you need to use this software.

Installation notes

Installation of IDL 8.5 on Sharc was a binary installation. IDL 8.5 was installed using the install_idl.sh script;the module file is /usr/local/modulefiles/apps/idl/8.5/binary.

ImageMagick

ImageMagick


http://www.exelisvis.co.uk/ProductsServices/IDL.aspx

http://www.exelisvis.com/docs/using_idl_home.html


Version 6.7.8.9-15

URL https://legacy.imagemagick.org/script/index.php

ImageMagick is a collection of tools for creating, editing, viewing, composing and converting images. Many bitmapand vector-based file formats are supported.

ImageMagick is installed on the cluster’s worker nodes. It provides a number of command-line programs plus an APIfor programmatic control.

Command-line usage

Some of the some of the most useful included command-line programs are:

display

Display an image stored in a file:

display winterview.jpg

display -resize 800x600 winterview.jpg

Display a series of files as a slide show:

display -delay 2 *.jpg

convert

Convert an image from one format to another e.g.

convert -format tiff pic1.jpg pic1.tiff

To capture from screen to a file:

convert -format jpeg X: newpicture.jpg

animate

Create an animation effect from a set of files containing snap-shots:

animate pic1.tiff pic2.tiff pic3.tiff pic4.tiff pic5.tiff

Create an animated-gif from a series of gif files (mypics*.gif):

convert -delay 40 -loop 0 mypics*.gif myanimated.gif

Note that non-gif files should first be converted:

convert -format gif mypic1.jpg mypic1.gif

2.3. ShARC 271

https://legacy.imagemagick.org/script/index.php


identify

View image metadata:

identify parkwood.jpg

More detailed information:

identify -verbose parkwood.jpg

More information on on these and other provided command-line tools (compare, composite, conjure, import,mogrify, montage and stream) can be found in the official ImageMagick 6 documentation.

Programmatic access (API)

There are ImageMagick APIs for most common programming languages, allowing you to script/automate your imagemanipulation using C++, Ruby, Perl, Python etc.

A list of ImageMagick APIs can be found on the ImageMagick 6 site. This list may not be comprehensive.

Installation notes

Version 6.7.8.9-15

This was installed on the worker nodes using the operating system’s package manager (i.e. the RPM for Centos 7.x isinstalled).

R (Intel Build)

R



R is a statistical computing language. This version of R is built using the Intel compilers and the Intel Math KernelLibrary. This combination can result in significantly faster runtimes in certain circumstances. For more details ofwhat is made faster, see this blog post which was written back when we built this version of R for the first time onSheffield’s old HPC system, Iceberg.

Most R extensions are written and tested for the gcc suite of compilers so it is recommended that you perform testingbefore switching to this version of R.

Loading the modules

After connecting to Sharc (see Establishing a SSH connection), start an interactive session with the qrshx command.

There are two types of the Intel builds of R, sequential and parallel.

sequential makes use of one CPU core and can be used as a drop-in replacement for the standard version of R installedon Sharc.


https://legacy.imagemagick.org/script/command-line-tools.php

https://legacy.imagemagick.org/script/develop.php



http://rse.shef.ac.uk/blog/intel-R-iceberg/


module load apps/R/3.4.0/intel-17.0-sequentialmodule load apps/R/3.3.2/intel-17.0-sequential

The parallel version makes use of multiple CPU cores for certain linear algebra routines since it is linked to the parallelversion of the Intel MKL. Note that only linear algebra routines are automatically parallelised.

module load apps/R/3.4.0/intel-17.0-parallelmodule load apps/R/3.3.2/intel-17.0-parallel

When using the parallel module, you must also ensure that you set the bash environment variableOMP_NUM_THREADS to the number of cores you require and also use the smp parallel environment. E.g. Addthe following to your submission script

#$ -pe smp 8export OMP_NUM_THREADS=8

module load apps/R/3.4.0/intel-17.0-parallel


To ensure that the Intel builds do not contaminate the standard gcc builds, the Intel R module files set the environmentvariable R_LIBS_USER to point to ~/R/intel_R/parallel-3.4.0/ or ~/R/intel_R/sequential-3.4.0/ respectively for version3.4.0 with similar paths for other versions.

As a user, you should not need to worry about this detail and just install packages as you usually would from withinR. e.g.

install.packages("dplyr")

The Intel builds of R will use the Intel compiler suite, instead of gcc and gfortran, to compile any C++ or Fortranextensions. The Intel build of R will ignore any packages installed in your home directory for the standard version ofR and vice versa

Installation Notes


version 3.4.0

This was a scripted install. It was compiled from source with Intel Compiler 17.0.0 and with –enable-R-shlib enabled.It was run in batch mode.

• intel-17.0-parallel.sh Downloads, compiles, tests and installs R 3.4.0 using Intel Compilers and theparallel MKL.

• intel-17.0-sequential.sh Downloads, compiles, tests and installs R 3.4.0 using Intel Compilers andthe sequential MKL.

• intel-17.0-parallel Parallel Module File

• intel-17.0-sequential Sequential Module File

version 3.3.2

This was a scripted install. It was compiled from source with Intel Compiler 17.0.0 and with –enable-R-shlib enabled.It was run in batch mode.

2.3. ShARC 273


• intel-17.0-parallel.sh Downloads, compiles, tests and installs R 3.3.2 using Intel Compilers and theparallel MKL.

• intel-17.0-sequential.sh Downloads, compiles, tests and installs R 3.3.2 using Intel Compilers andthe sequential MKL.

• intel-17.0-parallel Parallel Module File

• intel-17.0-sequential Sequential Module File

ITK-SNAP

ITK-SNAP

Version 3.6.0-rc1

Dependancies

URL http://www.itksnap.org

Documentation http://www.itksnap.org/pmwiki/pmwiki.php?n=Documentation.SNAP3

ITK-SNAP is a software application used to segment structures in 3D medical images.

Usage

The lastest version of ITK-SNAP can be activated using the module file:

module load apps/itksnap

Alternatively, you can load a specific version of ITK-SNAP:

module load apps/itksnap/3.6

Installation notes

ITK-SNAP was compiled using the install_itksnap.sh script, the module file is 3.6.

Java

Java

Latest Version 1.8.0_102

URL https://www.java.com/en/download/

Java is a programming language and computing platform first released by Sun Microsystems in 1995.


http://www.itksnap.org

http://www.itksnap.org/pmwiki/pmwiki.php?n=Documentation.SNAP3

https://www.java.com/en/download/


Interactive Usage

After connecting to ShARC (see Establishing a SSH connection), start an interactive session with the qrsh or qrshxcommand.

The default version of Java (which is also the most recent version; currently 1.8.0_102) is made available with thecommand

module load apps/java

Alternatively, you can explicitly load this version using:

module load apps/java/jdk1.8.0_102/binary

Explicitly loading a version will become more useful once multiple versions of Java are installed on this system.

Check that you have the version you expect. First, the runtime

$ java -version

java version "1.8.0_102"Java(TM) SE Runtime Environment (build 1.8.0_102-b14)Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

Now, the compiler

$ javac -version

javac 1.8.0_102

Installation notes


Java 1.8.0_102

1. Download Java SE Development Kit 8u102 from Oracle. Select the tarball (jdk-8u102-linux-x64.tar.gz) for Linux and the x64 CPU architecture family.

2. Save this file to /usr/local/media/java/1.8.0_102/.

3. Install Java using this script.

4. Install this modulefile as /usr/local/modulefiles/apps/java/jdk1.8.0_102/binary

MATLAB

MATLAB

Versions 2016a, 2016b, 2017a, 2017b

Support Level FULL

Dependancies None

URL http://uk.mathworks.com/products/matlab

2.3. ShARC 275

http://www.oracle.com/technetwork/java/javase/downloads

http://uk.mathworks.com/products/matlab


Documentation http://uk.mathworks.com/help/matlab

Scientific computing and visualisation.

Interactive Usage

After connecting to ShARC (see Establishing a SSH connection), start an interactive session with the qrshx com-mand.

The latest version of MATLAB (currently 2017b) is made available by running:

module load apps/matlab

Alternatively, you can load a specific version with one of the following commands:

module load apps/matlab/2016a/binarymodule load apps/matlab/2016b/binarymodule load apps/matlab/2017a/binarymodule load apps/matlab/2017b/binary

You can then run MATLAB by entering matlab


Here, we assume that you wish to run the program helloworld.m on the system:

function helloworlddisp('Hello World!')

end

First, you need to write a batch submission file. We assume you’ll call this my_job.sge:

#!/bin/bash#$ -l rmem=4G # Request 4 GB of real memory#$ -cwd # Run job from current directorymodule load apps/matlab/2017b/binary # Make specific version of MATLAB available

matlab -nodesktop -nosplash -r helloworld

Ensuring that helloworld.m and my_job.sge are both in your current working directory, submit your job to thebatch system:

qsub my_job.sge

Note that we are running the script helloworld.m but we drop the .m in the call to MATLAB. That is, we do -rhelloworld rather than -r helloworld.m. The output will be written to the job .o file when the job finishes.

MATLAB Compiler and running free-standing compiled MATLAB programs

The MATLAB compiler mcc can be used to generate standalone executables. These executables can then be run onother computers that does not have MATLAB installed. We strongly recommend you use R2016b or later versions totake advantage of this feature.

To compile a MATLAB function or script for example called myscript.m the following steps are required:


http://uk.mathworks.com/help/matlab


# Load the matlab 2017b modulemodule load apps/matlab/2017b/binary

# Compile your program to generate the executable myscript and# also generate a shell script named run_myscript.shmcc -m myscript.m

# Finally run your program./run_myscript.sh $MCRROOT

If myscript.m is a MATLAB function that require inputs these can be suplied on the command line. For exampleif the first line of myscript.m reads:

function out = myscript ( a , b , c )

then to run it with 1.0, 2.0, 3.0 as its parameters you will need to type:

./run_myscript.sh $MCRROOT 1.0 2.0 3.0

After a successful compilation and running you can transfer your executable and the runscript to another computer.That computer does not have to have MATLAB installed or licensed on it but it will have to have the MATLAB runtimesystem installed. This can be done by either downloading the MATLAB runtime environment from Mathworks website or by copying the installer file from the cluster itself which resides in:

$MCRROOT/toolbox/compiler/deploy/glnxa64/MCRInstaller.zip

This file can be unzipped in a temporary area and run the setup script that unzipping yields to install the MATLABruntime environment. Finally the environment variable $MCRROOT can be set to the directory containing the runtimeenvironment.

Parallel MATLAB

Parallel Matlab is currently restricted to single node operation (16 cores). An example batch scriptmy_parallel_job.sh is:

#!/bin/bash#$ -l rmem=2G#$ -pe smp 12module load apps/matlab/2017b/binary#Run parallel_example.mmatlab -nodisplay -r parallel_example

where parallel_example.m is:

%create parallel pool of workers on the local node%Ensure that this is the same number as what you requested from the schedulerpool = parpool('local',12)disp('serial time')ticn = 200;A = 500;a = zeros(n);for i = 1:n

a(i) = max(abs(eig(rand(A))));endtoc

2.3. ShARC 277


disp('Parallel time')ticn = 200;A = 500;a = zeros(n);parfor i = 1:n

a(i) = max(abs(eig(rand(A))));endtoc

delete(pool)

Note that multi-node parallel Matlab is not yet configured on this cluster. Task arrays are supported by all versions,however it is recommended that 2017a (or later) is used

Training

• CiCS run an Introduction to Matlab course

• In November 2015, CiCS hosted a masterclass in Parallel Computing in MATLAB. The materials are availableonline

Installation notes


Installation and configuration is a four-stage process:

• Set up the floating license server (the license server for earlier MATLAB versions can be used), ensuring that itcan serve licenses for any new versions of MATLAB that you want to install

• Run a graphical installer to download MATLAB archive files used by the main (automated) installation process

• Run the same installer in ‘silent’ command-line mode to perform the installation using those archive files and atext config file.

• Install a relevant modulefile

In more detail:

1. If necessary, update the floating license keys on licserv4.shef.ac.uk to ensure that the licenses areserved for the versions to install.

2. Log on to Mathworks site to download the MATLAB installer package for 64-bit Linux ( for R2017b this wascalled matlab_R2017b_glnxa64.zip )

3. unzip the installer package in a directory with ~10GB of space (needed as many MATLAB archive fileswill subsequently be downloaded here). Using a directory on an NFS mount (e.g. /data/${USER}/MathWorks/R2017b) allows the same downloaded archives to be used to install MATLAB on multipleclusters.

4. ./install to start the graphical installer (needed to download the MATLAB archive files).

5. Select install choice of Log in to Mathworks Account and log in with a License Administrator account (not aLicensed End User (personal) account).

6. Select Download only.


http://rcg.group.shef.ac.uk/courses/matlab/




7. Select the offered default Download path and select the directory you ran ./install from. Wait a while forall requested archive files to be downloaded.

8. Next, ensure installer_input.txt looks like the following

fileInstallationKey=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-→˓XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX-XXXXXagreeToLicense=yesoutputFile=matlab_2017b_install.logmode=silentlicensePath=/usr/local/packages/matlab/network.liclmgrFiles=falselmgrService=false

9. Create the installation directory e.g.:

mkdir -m 2755 -p /usr/local/packages/apps/matlab/R2017b/binarychown ${USER}:app-admins /usr/local/packages/apps/matlab/R2017b/binary

10. Run the installer using our customized installer_input.txt like so: ./install -mode silent-inputFile ${PWD}/installer_input.txt ; installation should finish with exit status 0 if all hasworked.

11. Ensure the contents of the install directory and the modulefile are writable by those in app-admins groupe.g.:

chmod -R g+w ${USER}:app-admins /usr/local/packages/apps/matlab/R2017b /usr/local/→˓modulefiles/apps/matlab/2017b

The modulefile is /usr/local/modulefiles/apps/matlab/2017b/binary.

Molpro

Molpro

Version 2015.1.22

Dependencies No additional modules loaded.

URL http://www.molpro.net/

Documentation http://www.molpro.net/info/users?portal=user

Molpro is a comprehensive system of ab initio programs for advanced molecular electronic structure calculations,designed and maintained by H.-J. Werner and P. J. Knowles, and containing contributions from many other authors.

Usage

Molpro 2015.1.22 can be activated using the module file:

module load apps/molpro/2015.1.22/binary

The Molpro executable is molpro.

Important note: Only licensed users of Molpro are entitled to use the code; refer to Molpro’s website for li-cense details: http://www.molpro.net/info/products.php?portal=visitor&choice=Licence+types. Access to Molpro

2.3. ShARC 279

http://www.molpro.net/

http://www.molpro.net/info/users?portal=user

http://www.molpro.net/info/products.php?portal=visitor&choice=Licence+types


on ShARC is restricted to members of the unix group molpro. To be added to this group, please [email protected] and provide evidence of your eligibility to use Molpro.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run molpro and which is submitted to the queue by typing qsub my_job.sh.


module load apps/molpro/2015.1.22/binary

molpro -n 4 my_input.inp

The script requests 4 cores using the Open MPI parallel environment mpi with a runtime of 30 mins and 2 GB of realmemory per core. The Molpro input file is my_input.inp.

Installation notes

Molpro 2015.1.22 was installed as a binary installation using the install_molpro.sh script; the module file is/usr/local/modulefiles/apps/molpro/2015.1.22/binary.

The Molpro 2015.1.22 installation was tested by running a batch job using the my_job.sh batch script, above, and anexample input file from Molpro’s quick-start website (http://www.molpro.net/info/2015.1/doc/quickstart/index.html).

OpenFOAM

OpenFOAM

Version 4.1

Dependencies GCC compiler and Open MPI; Scotch/PT-Scotch v6.0.3

URL https://openfoam.org

Documentation https://cfd.direct/openfoam/user-guide/

OpenFOAM is leading software for computational fluid dynamics (CFD). It is licensed free and open source onlyunder the GNU General Public Licence (GPL) by the OpenFOAM Foundation.

Usage

OpenFOAM 4.1 can be activated using the module file:

module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1


http://www.molpro.net/info/2015.1/doc/quickstart/index.html

https://openfoam.org

https://cfd.direct/openfoam/user-guide/


Installation notes

OpenFOAM 4.1 was installed using the install_openfoam.sh script, the module file is gcc-6.2-openmpi-2.0.1. The following optional dependencies were built as part of the installation process: Scotch/PT-Scotch v6.0.3 (located in /usr/local/packages/apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1/ThirdParty-4.1). The follow-ing optional dependencies were not installed: ParaView and CGAL.

Installation was tested as follows (Getting Started example from https://openfoam.org/download/4-1-source/):

$ mkdir /data/$USER/tests/openfoam/run$ cd /data/$USER/tests/openfoam/run$ module load apps/openfoam/4.1/gcc-6.2-openmpi-2.0.1$ cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .$ cd pitzDaily$ blockMesh$ simpleFoam$ paraFoam

ORCA

ORCA

Version 4.0.1

Dependencies Module for Open MPI 2.1.1 loaded. Open MPI 2.0.2 and above is required for theORCA 4.0.1 MPI binaries.

URL https://orcaforum.cec.mpg.de/

Documentation https://cec.mpg.de/fileadmin/media/Forschung/ORCA/orca_manual_4_0_1.pdf

An ab initio, DFT and semiempirical SCF-MO package. The program ORCA is a modern electronic structure programpackage written by F. Neese, with contributions from many current and former coworkers and several collaboratinggroups. The binaries of ORCA are available free of charge for academic users for a variety of platforms.

Usage

ORCA 4.0.1 can be activated using the module file:

module load apps/orca/4.0.1/binary

The ORCA 4.0.1 executable is orca. All the ORCA 4.0.1 executables are in /usr/local/packages/apps/orca/4.0.1/binary/orca_4_0_1_linux_x86-64_openmpi202.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run orca and which is submitted to the queue by typing qsub my_job.sh.

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00

2.3. ShARC 281

https://openfoam.org/download/4-1-source/


https://cec.mpg.de/fileadmin/media/Forschung/ORCA/orca_manual_4_0_1.pdf


#$ -l rmem=2G

module load apps/orca/4.0.1/binary

orca my_input.inp > my_input.out

The script requests a serial job with a runtime of 30 mins and 2G of real memory. The ORCA 4.0.1 input file ismy_input.inp.

Installation notes

ORCA 4.0.1 was installed as a binary installation using the install_orca.sh script; the module file is binary.

The ORCA 4.0.1 installation was tested by running a batch job using the my_job.sh batch script, above, with thetext, below, in an ORCA input file.

## My first ORCA calculation :-)#! HF SVP

* xyz 0 1C 0 0 0O 0 0 1.13

*

PLUMED

PLUMED

Version 2.3.2

Dependencies GCC compiler and Open MPI. Modules loaded for GCC 6.2.0 and Open MPI 2.1.1.

URL http://www.plumed.org

Documentation https://plumed.github.io/doc-v2.3/user-doc/html/index.html

PLUMED is an open source library for free energy calculations in molecular systems which works together with someof the most popular molecular dynamics engines. Free energy calculations can be performed as a function of manyorder parameters with a particular focus on biological problems, using state of the art methods such as metadynamics,umbrella sampling and Jarzynski-equation based steered MD. The software, written in C++, can be easily interfacedwith both fortran and C/C++ codes.

Usage

PLUMED 2.3.2 can be activated using the module file:

module load apps/plumed/2.3.2/gcc-6.2-openmpi-2.1.1

The PLUMED executable is plumed.


http://www.plumed.org

https://plumed.github.io/doc-v2.3/user-doc/html/index.html


Installation notes

PLUMED 2.3.2 was installed using the install_plumed.sh script; the module file is gcc-6.2-openmpi-2.1.1.

The installation of PLUMED 2.3.2 was tested using the make command as part of running the above installationscript.

Polyrate

Polyrate (with Gaussrate)

Version 17 (20170808)

Dependencies Modules for Open MPI 2.1.1, GCC 6.2.0 and Gaussian 09 D.01 loaded.

URL for Polyrate https://comp.chem.umn.edu/polyrate/

URL for Gaussrate https://t1.chem.umn.edu/gaussrate/

Documentation for Polyrate https://comp.chem.umn.edu/polyrate/170808_Polyrate_Manual_v17.pdf

Documentation for Gaussrate https://comp.chem.umn.edu/gaussrate/170808_Gaussrate_17B_Manual.pdf

Quickstart Guide for Gaussrate https://comp.chem.umn.edu/gaussrate/GAUSSRATEQuickstartGuide.pdf

Polyrate 17: Computer Program for the Calculation of Chemical Reaction Rates for Polyatomics. Gaussrate 17-Bprovides an interface between two other programs: Polyrate version 2017 and Gaussian 16 / 09. Note: Polyrate 17and Gaussrate 17-B were compiled on ShARC to use the Reaction Path Variational Transition State Theory (RP-VTST)executables.

Usage

Polyrate 17 and Gaussrate 17-B can be activated using the module file:

module load apps/polyrate/17_20170808/gaussrate17-B/gcc-6.2-openmpi-2.1.1

The Polyrate/Gaussrate executable script is shuttle. All the Polyrate 17 and Gaussrate 17-B ex-ecutables are in /usr/local/packages/apps/polyrate/17_20170808/gaussrate17-B/gcc-6.2-openmpi-2.1.1/polyrate17/exe.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, which is submitted to the queue by typing qsub my_job.sh.

#!/bin/bash#$ -cwd#$ -l h_rt=00:30:00#$ -l rmem=2G

2.3. ShARC 283

https://comp.chem.umn.edu/polyrate/

https://t1.chem.umn.edu/gaussrate/

https://comp.chem.umn.edu/polyrate/170808_Polyrate_Manual_v17.pdf

https://comp.chem.umn.edu/polyrate/170808_Polyrate_Manual_v17.pdf

https://comp.chem.umn.edu/gaussrate/170808_Gaussrate_17B_Manual.pdf

https://comp.chem.umn.edu/gaussrate/170808_Gaussrate_17B_Manual.pdf

https://comp.chem.umn.edu/gaussrate/GAUSSRATEQuickstartGuide.pdf

https://comp.chem.umn.edu/gaussrate/GAUSSRATEQuickstartGuide.pdf


#$ -pe smp 4

module load apps/polyrate/17_20170808/gaussrate17-B/gcc-6.2-openmpi-2.1.1

export GAUSS_SCRDIR=$TMPDIRexport scratchdir=$TMPDIR

./ch5tr1.jc

The script requests 4 cores using the shared memory parallel environment smp with a runtime of 30 mins and 2 GBof real memory per core. The Polyrate job control script is ch5tr1.jc and is for the test run example ch5tr1. Note:The Polyrate environment variables polydir and gausspath are set in the module file.

The ch5tr1 test run, an example calculation using Gaussrate, is described in the documentation and the inputfiles and job control script can be found at /usr/local/packages/apps/polyrate/17_20170808/gaussrate17-B/gcc-6.2-openmpi-2.1.1/polyrate17/gaussrate17-B/testrun/ch5.

Installation notes

Polyrate 17 and Gaussrate 17-B were installed using the install_polyrate17.sh script; the mod-ule file is /usr/local/modulefiles/apps/polyrate/17_20170808/gaussrate17-B/gcc-6.2-openmpi-2.1.1. Polyrate 17 and Gaussrate 17-B were compiled on ShARC to use the Reaction Path VariationalTransition State Theory (RP-VTST) executables.

The Polyrate 17 and Gaussrate 17-B installation was tested by running a batch job using the my_job.sh batch script,above, for the ch5tr1 test run.

Python

Python

Support Level Core

URL https://python.org

This page documents the “miniconda” installation on ShARC. This is the recommended way of using Python, and thebest way to be able to configure custom sets of packages for your use.

“conda” a Python package manager, allows you to create “environments” which are sets of packages that you canmodify. It does this by installing them in your home area. This page will guide you through loading conda and thencreating and modifying environments so you can install and use whatever Python packages you need.

Using conda Python

After connecting to ShARC (see Establishing a SSH connection), start an interactive session with the qrshx or qrshcommand.

Conda Python can be loaded with:

module load apps/python/conda


https://python.org


The root conda environment (the default) provides Python 3 and no extra modules, it is automatically updated, andnot recommended for general use, just as a base for your own environments. There is also a python2 environment,which is the same but with a Python 2 installation.

Quickly Loading Anaconda Environments

There are a small number of environments provided for everyone to use, these are the default root and python2environments as well as various versions of Anaconda for Python 3 and Python 2.

The anaconda environments can be loaded through provided module files:

module load apps/python/anaconda2-4.2.0module load apps/python/anaconda3-4.2.0

Where anaconda2 represents Python 2 installations and anaconda3 represents Python 3 installations. Thesecommands will also load the apps/python/conda module and then activate the anaconda environment specified.

Note: Anaconda 2.5.0 and higher are compiled with Intel MKL libraries which should result in higher numericalperformance.

Using conda Environments

Once the conda module is loaded you have to load or create the desired conda environments. For the documentationon conda environments see the conda documentation.

You can load a conda environment with:

source activate python2

where python2 is the name of the environment, and unload one with:

source deactivate

which will return you to the root environment.

It is possible to list all the available environments with:

conda env list

Provided system-wide are a set of anaconda environments, these will be installed with the anaconda version numberin the environment name, and never modified. They will therefore provide a static base for derivative environments orfor using directly.

Creating an Environment

Every user can create their own environments, and packages shared with the system-wide environments will not bereinstalled or copied to your file store, they will be symlinked, this reduces the space you need in your /homedirectory to install many different Python environments.

To create a clean environment with just Python 2 and numpy you can run:

conda create -n mynumpy python=2.7 numpy

2.3. ShARC 285

http://conda.pydata.org/docs/using/envs.html


This will download the latest release of Python 2.7 and numpy, and create an environment named mynumpy.

Any version of Python or list of packages can be provided:

conda create -n myscience python=3.5 numpy=1.8.1 scipy

If you wish to modify an existing environment, such as one of the anaconda installations, you can clone that envi-ronment:

conda create --clone anaconda3-4.2.0 -n myexperiment

This will create an environment called myexperiment which has all the anaconda 4.2.0 packages installed withPython 3.

Installing Packages Inside an Environment

Once you have created your own environment you can install additional packages or different versions of packagesinto it. There are two methods for doing this, conda and pip, if a package is available through conda it is stronglyrecommended that you use conda to install packages. You can search for packages using conda:

conda search pandas

then install the package using:

conda install pandas

if you are not in your environment you will get a permission denied error when trying to install packages, if thishappens, create or activate an environment you own.

If a package is not available through conda you can search for and install it using pip, i.e.:

pip search colormath

pip install colormath

Using Python with MPI

There is an experimental set of packages for conda that have been compiled by the RCG team, which allow you to usea MPI stack entirely managed by conda. This allows you to easily create complex evironments and use MPI withoutworrying about other modules or system libraries.

To get access to these packages you need to run the following command to add the repo to your conda config:

conda config --add channels file:///usr/local/packages/apps/conda/conda-bld/

you should then be able to install the packages with the openmpi feature, which currently include openmpi, hdf5,mpi4py and h5py:

conda create -n mpi python=3.5 openmpi mpi4py

Currently, there are Python 2.7 and 3.5 versions of mpi4py and h5py compiled in this repository.

The build scripts for these packages can be found in this GitHub repository.


https://github.com/rcgsheffield/conda-packages


Installation Notes


The conda package manager is installed in /usr/share/packages/apps/conda, it was installed using theminiconda installer.

It is important to regularly update the root environment to keep the conda package manager up to date. To do thislogin as a sa_ account (with write permissions to /usr/local/packages/apps/conda) and run:

$ conda update --all$ conda update conda

Installing a New Version of Anaconda

Run the following as a sa_ user (with write permissions to /usr/local/packages/apps/conda:

$ conda create -n anaconda3-<VERSION> python=3 anaconda=<VERSION>$ conda create -n anaconda2-<VERSION> python=2 anaconda=<VERSION>

Then copy the modulefile for the previous version of anaconda to the new version and update the name of the environ-ment. Also you will need to append the new module to the conflict line in apps/python/.conda-environments.tcl.

Singularity

Singularity

Version 2.3.2

URL http://singularity.lbl.gov/

Designed around the notion of extreme mobility of compute and reproducible science, Singularity enables users tohave full control of their operating system environment. This means that a non-privileged user can “swap out” theoperating system on the host for one they control. So if the host system is running Centos Linux but your applicationruns in Ubuntu Linux, you can create an Ubuntu image, install your applications into that image, copy the image toanother host, and run your application on that host in it’s native Ubuntu environment.

Singularity also allows you to leverage the resources of whatever host you are on. This includes high-speed clusterinterconnects, resource managers, file systems, GPUs and/or accelerators, etc.

About Singularity Containers (Images)

Similar to Docker, a Singularity container (image) is a self-contained software stack. As Singularity does not requirea root-level daemon to run its images it is compatible for use with ShARC’s scheduler inside your job scripts. Therunning images also uses the credentials of the person calling it.

In practice, this means that an image created on your local machine with all your research software installed for localdevelopment will also run on the ShARC cluster.

Pre-built images have been provided on the cluster and can also be download for use on your local machine (seeInteractive Usage of Singularity Images). Creating and modifying images however, requires root permission and somust be done on your machine (see Creating Your Own Singularity Images).

2.3. ShARC 287

http://conda.pydata.org/miniconda.html

http://singularity.lbl.gov/


Singularity images are currently provided for:

• Caffe

• Theano

• Torch

• Tensorflow

Interactive Usage of Singularity Images

To use Singularity interactively, an interactive session must first be requested using qrshx for example.

To get an interactive shell in to the image, use the following command:

singularity shell path/to/imgfile.img

Or if you prefer bash:

singularity exec path/to/imgfile.img /bin/bash

Note that the exec command can also be used to execute other applications/scripts inside the image or from themounted directories (See Automatic Mounting of ShARC Filestore Inside Images):

singularity exec path/to/imgfile.img my_script.sh

Note: You may get a warning similar to:

groups: cannot find name for group ID ...

This can be ignored and will not have an affect on running the image.

Submitting Batch Jobs That Uses Singularity Images

When submitting a job that uses a Singularity image, it is not possible to use the interactive shell (e.g. singularityshell or singularity exec path/to/imgfile.img /bin/bash). You must use the exec commandto call the desired application or script directly.

For example, if we would like to use a command ls / to get the content of the root folder in the image, two approachesare shown in the following job script my_singularity_job.sh:

#!/bin/bash#$ -l rmem=8G# We requested 8GB of memory in the line above, change this according to your# needs e.g. add -l gpu=1 to reqest a single GPU

#Calling ls directly using the exec commandsingularity exec path/to/imgfile.img ls /

#Have Singularity call a custom script from your home or other mounted directories#Don't forget to make the script executable before running by using chmodchmod +x ~/myscript.shsingularity exec path/to/imgfile.img ~/myscript.sh



Where the content of ~/myscript.sh is shown below:

#!/bin/bash

ls /

The job can then be submitted as normal with qsub:

qsub my_singularity_job.sh

Using Nvidia GPU with Singularity Images

You can use GPUs in your image by adding the --nv flag to the command e.g. for running interactively:

singularity shell --nv myimage.simg

or when running within the batch file:

singularity exec --nv myimage.sim myscript.sh

A quick way to test that GPU is enabled in your image is by running the command:

nvidia-smi

Where you will get something similar to the following:

Tue Mar 28 16:43:08 2017+-----------------------------------------------------------------------------+| NVIDIA-SMI 367.57 Driver Version: 367.57 ||-------------------------------+----------------------+----------------------+| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. ||===============================+======================+======================|| 0 GeForce GTX TITAN Off | 0000:01:00.0 On | N/A || 30% 35C P8 18W / 250W | 635MiB / 6078MiB | 1% Default |+-------------------------------+----------------------+----------------------+

Automatic Mounting of ShARC Filestore Inside Images

When running Singularity images on ShARC, the paths /fastdata, /data, /home, /scratch, /shared areautomatically mounted to your ShARC directories.

Image Index on Github

All Singularity container definitions available on ShARC can be found at https://github.com/rses-singularity. Thedefinition files can be used as a template for building your own images.

Installing Singularity on Your Local Machine

You will need Singularity installed on your machine in order to locally run, create and modify images.

The following is the installation command for debian/ubuntu based systems:

2.3. ShARC 289

https://github.com/rses-singularity


#Updating repository and installing dependenciessudo apt-get update && \sudo apt-get install \python \dh-autoreconf \build-essential

# Installing Singularitygit clone https://github.com/singularityware/singularity.gitcd singularity./autogen.sh./configure --prefix=/usr/local --sysconfdir=/etcmakesudo make install

Manually mounting paths

When using ShARC’s pre-built images on your local machine, it may be useful to mount the existing directories in theimage to your own path. This can be done with the flag -B local/path:image/path with the path outside ofthe image left of the colon and the path in the image on the right side, e.g.

singularity shell -B local/datapath:/data,local/fastdatapath:/fastdata path/to/→˓imgfile.img

The command mounts the path local/datapath on your local machine to the /data path in the image. Multiplemount points can be joined with , as shown above where we additionally specify that local/fastdata mounts to/fastdata. The /home folder is automatically mounted by default.

Note: In order to mount a path, the directory must already exist within the image.

Creating Your Own Singularity Images

Root access is required for modifying Singularity images so if you need to edit an image it must be done on yourlocal machine. However you can create disk images and import docker images using normal user privileges onrecent versions of Singularity

First create a Singularity definition file for bootstrapping an image your image. An example definition file we’ll nameSingularity is shown below

Bootstrap: dockerFrom: ubuntu:latest

%setup#Runs on host. The path to the image is $SINGULARITY_ROOTFS

%post#Post setup, runs inside the image

#Default mount pathsmkdir /scratch /data /shared /fastdata

#Install the packages you needapt-get install git vim cmake



%runscript#Runs inside the image every time it starts up

%test#Test script to verify that the image is built and running correctly

The definition file takes a base image from docker hub, in this case the latest version of Ubuntu ubuntu:latest.Other images on the hub can also be used as the base for the Singularity image, e.g. From: nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 uses Nvidia’s docker image with Ubuntu 16.04 that already has CUDA 8 in-stalled.

After creating a definition file, use the build command to build the image from your definition file:

sudo singularity build myimage.simg Singularity

It is also possible to build Singularity images directory from Singularity hub or docker hub:

#Singularity hubsudo singularity build myimage.simg shub://GodloveD/ubuntu:latest

#Docker hubsudo singularity build myimage.simg docker://ubuntu:latest

By default, the build command creates a read-only squashfs file. It is possible to add the --writable or--sandbox flag to the build command in order to create a writable ext image or a writable sandbox directoryrespectively.

sudo singularity build --sandbox myimage_folder Singularity

You will also need to add the --writable flag to the command when going in to change the contents of an image:

sudo singularity shell --writable myimage_folder

How Singularity is installed and ‘versioned’ on the cluster

Singularity, unlike much of the other key software packages on ShARC, is not activated using module files. This isbecause module files are primarily for the purpose of being able to install multiple version of the same software and forsecurity reasons only the most recent version of Singularity is installed. The security risks associated with providingoutdated builds of Singularity are considered to outweigh the risk of upgrading to backwards incompatible versions.

Singularity has been installed on all worker nodes using the latest RPM package from the EPEL repository.

spark

Spark

Version 2.1

URL http://spark.apache.org/

Apache Spark is a fast and general engine for large-scale data processing.

2.3. ShARC 291

https://hub.docker.com/

https://singularity-hub.org/

https://hub.docker.com/

https://fedoraproject.org/wiki/EPEL

http://spark.apache.org/


Interactive Usage

After connecting to Sharc (see Establishing a SSH connection), start an interactive session with the qrsh or qrshxcommand.

Before using Spark, you will need to load a version of Java. For example


To make Spark available, use the following module command

module load apps/spark/2.1.0/gcc-4.8.5

You can now start a Spark shell session with

spark-shell

SparkR

To use SparkR, you will additionally need to load a version of R

module load apps/java/jdk1.8.0_102/binarymodule load apps/spark/2.1.0/gcc-4.8.5module load apps/R/3.3.2/gcc-4.8.5

Now you can start a SparkR session

sparkR

R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"Copyright (C) 2016 The R Foundation for Statistical ComputingPlatform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.You are welcome to redistribute it under certain conditions.Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.Type 'contributors()' for more information and'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or'help.start()' for an HTML browser interface to help.Type 'q()' to quit R.

Setting the number of cores

The installation of Spark on Sharc is limited to jobs that make use of one node. As such, the maximum number ofCPU cores you can request for a Spark job is limited to 16 cores.

First, you must request cores from the scheduler. That is, you add the following line to your submission script torequest 4 cores



#$ -pe smp 4

You must also tell Spark to only use 4 cores by setting the MASTER environment variable

export MASTER=local[4]

A full example is given at https://github.com/mikecroucher/HPC_Examples/tree/master/languages/Python/pyspark_pi

Installation notes


Spark 2.0 was built using the system gcc 4.8.5

qrsh -l rmem=10G

module load apps/java/jdk1.8.0_102/binarytar -xvzf ./spark-2.1.0.tgzcd spark-2.1.0

mkdir -p /usr/local/packages/apps/spark/2.1cd ..mv spark-2.1.0 /usr/local/packages/apps/spark/2.1

The default install of Spark is incredibly verbose. Even a ‘Hello World’ program results in many lines of [INFO]. Tomake it a little quieter, the default log4j level has been reduced from INFO to WARN:

cd /usr/local/packages/apps/spark/2.1/spark-2.1.0/conf/cp log4j.properties.template log4j.properties

The file log4j.properties was then edited so that the line beginning log4j.rootCategory reads:

log4j.rootCategory=WARN, console

Modulefile

Version 2.1

The following module file is on the system at /usr/local/modulefiles/apps/spark/2.1.0/gcc-4.8.5

#%Module1.0######################################################################### Spark module file##


set sparkhome /usr/local/packages/apps/spark/2.1/spark-2.1.0

# Use only one core. User can override this if they wantsetenv MASTER local\[1\]setenv SPARK_HOME $sparkhomeprepend-path PATH $sparkhome/bin

2.3. ShARC 293

https://github.com/mikecroucher/HPC_Examples/tree/master/languages/Python/pyspark_pi


SU2

SU2

Version 5.0.0

Dependencies Modules loaded for GCC compiler 6.2.0 and Open MPI 2.1.1

URL https://su2code.github.io/

Documentation https://github.com/su2code/SU2/wiki

The SU2 suite is an open-source collection of C++ based software tools for performing Partial Differential Equation(PDE) analysis and solving PDE-constrained optimization problems. The toolset is designed with Computational FluidDynamics (CFD) and aerodynamic shape optimization in mind, but is extensible to treat arbitrary sets of governingequations such as potential flow, elasticity, electrodynamics, chemically-reacting flows, and many others. SU2 isunder active development by the Aerospace Design Lab (ADL) of the Department of Aeronautics and Astronautics atStanford University and many members of the community, and is released under an open-source license.

Usage

SU2 5.0.0 can be activated using the module file:

module load apps/su2/5.0.0/gcc-6.2-openmpi-2.1.1

Installation notes

SU2 5.0.0 was installed using the install_su2.sh script; the module file is /usr/local/modulefiles/apps/su2/5.0.0/gcc-6.2-openmpi-2.1.1.

Taverna-cli

Taverna-cli

Version 2.5.0

Dependancies

URL http://www.taverna.org.uk

Documentation http://www.taverna.org.uk/documentation/taverna-2-x/

Taverna is an open source and domain-independent Workflow Management System – a suite of tools used to designand execute scientific workflows and aid in silico experimentation. Taverna-cli is the command line version (no GUI).

Usage

The lastest version of Taverna-cli can be activated using the module file:


https://su2code.github.io/

https://github.com/su2code/SU2/wiki

http://www.taverna.org.uk

http://www.taverna.org.uk/documentation/taverna-2-x/


module load apps/taverna/cli

Alternatively, you can load a specific version of Taverna-cli:

module load apps/taverna/cli/2.5.0

Installation notes

Taverna-cli was compiled using the install_taverna-cli.sh script, the module file is 2.5.0.

Tensorflow

Tensorflow

URL https://www.tensorflow.org/

TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in thegraph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors)communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs orGPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchersand engineers working on the Google Brain Team within Google’s Machine Intelligence research organization for thepurposes of conducting machine learning and deep neural networks research, but the system is general enough to beapplicable in a wide variety of other domains as well.

About Tensorflow on ShARC


Tensorlfow is available on ShARC as both Singularity images and by local installation.

As Tensorflow and all its dependencies are written in Python, it can be installed locally in your home directory. The useof Anaconda (Python) is recommended as it is able to create a virtual environment in your home directory, allowingthe installation of new Python packages without admin permission.

This software and documentation is maintained by the RSES group and GPUComputing@Sheffield. For featurerequests or if you encounter any problems, please raise an issue on the GPU Computing repository.

Tensorflow Singularity Images



#CPU Tensorflow/usr/local/packages/singularity/images/tensorflow/cpu.img

#GPU Tensorflow/usr/local/packages/singularity/images/tensorflow/gpu.img

2.3. ShARC 295

https://www.tensorflow.org/


http://gpucomputing.shef.ac.uk/




singularity exec --nv /usr/local/packages/singularity/images/tensorflow/gpu.img /bin/→˓bash


singularity exec --nv /usr/local/packages/singularity/images/tensorflow/gpu.img→˓python your_tensorflow_script.py




Tensorflow is installed as part of Anaconda and can be found inside the image at:

/usr/local/anaconda3-4.2.0/lib/python3.5/site-packages/tensorflow


Image Index



– CPU

* /usr/local/packages/singularity/images/tensorflow/cpu.img

– GPU

* usr/local/packages/singularity/images/tensorflow/gpu.img

• CPU Images

– Latest: 1.0.1-CPU-Ubuntu16.04-Anaconda3.4.2.0.img (GCC 5.4.0, Python 3.5)

* Path: /usr/local/packages/singularity/images/tensorflow/1.0.1-CPU-Ubuntu16.04-Anaconda3.4.2.0.img

* Def file: /sharc/software/apps/singularity/tensorflow_cpu.def

• GPU Images

– Latest: 1.0.1-GPU-Ubuntu16.04-Anaconda3.4.2.0-CUDA8-cudNN5.0.img (GCC 5.4.0, Python 3.5)

* Path: /usr/local/packages/singularity/images/tensorflow/1.0.1-GPU-Ubuntu16.04-Anaconda3.4.2.0-CUDA8-cudNN5.0.img

* Def file: /sharc/software/apps/singularity/tensorflow_gpu.def



Installation in Home Directory

The following is an instruction on how to setup Tensorflow on your user account.


Load the relevant modules (our example uses CUDA 8.0 with cuDNN 5.1 but other versions are available)

module load apps/python/anaconda3-4.2.0module load libs/cudnn/6.0/binary-cuda-8.0.44

Create a conda environment to load relevant modules on your local user account and activate it

conda create -n tensorflow python=3.5source activate tensorflow

Then install tensorflow 1.4 for GPU with the following commands

pip install tensorflow-gpu

For CPU-only or other python versions, see the offical installation page to get the correct binaries.

You can test that Tensorflow is running on the GPU with the following python code

import tensorflow as tf# Creates a graph.with tf.device('/gpu:0'):

a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')c = tf.matmul(a, b)

# Creates a session with log_device_placement set to True.sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))# Runs the op.print(sess.run(c))

Which gives the following results

[[ 22. 28.][ 49. 64.]]


The previous instuctions installs Tensorflow and its dependencies inside your home directory but every time you usea new session or within your job scripts, the modules must be loaded and conda must be activated again. Use thefollowing command to activate the Conda environment with Tensorflow installed:

module load apps/python/anaconda3-4.2.0module load libs/cudnn/6.0/binary-cuda-8.0.44source activate tensorflow

Using multiple GPUs

Example taken from tensorflow documentation.

If you would like to run TensorFlow on multiple GPUs, you can construct your model in a multi-tower fashion whereeach tower is assigned to a different GPU. For example:

2.3. ShARC 297

https://www.tensorflow.org/install/install_linux

https://www.tensorflow.org/versions/r0.11/how_tos/using_gpu/index.html


import tensorflow as tf# Creates a graph.c = []for d in ['/gpu:2', '/gpu:3']:with tf.device(d):a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])c.append(tf.matmul(a, b))

with tf.device('/cpu:0'):sum = tf.add_n(c)

# Creates a session with log_device_placement set to True.sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))# Runs the op.print sess.run(sum)

You will see the following output.

Device mapping:/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: Tesla K20m, pci busid: 0000:02:00.0/job:localhost/replica:0/task:0/gpu:1 -> device: 1, name: Tesla K20m, pci busid: 0000:03:00.0/job:localhost/replica:0/task:0/gpu:2 -> device: 2, name: Tesla K20m, pci busid: 0000:83:00.0/job:localhost/replica:0/task:0/gpu:3 -> device: 3, name: Tesla K20m, pci busid: 0000:84:00.0Const_3: /job:localhost/replica:0/task:0/gpu:3Const_2: /job:localhost/replica:0/task:0/gpu:3MatMul_1: /job:localhost/replica:0/task:0/gpu:3Const_1: /job:localhost/replica:0/task:0/gpu:2Const: /job:localhost/replica:0/task:0/gpu:2MatMul: /job:localhost/replica:0/task:0/gpu:2AddN: /job:localhost/replica:0/task:0/cpu:0[[ 44. 56.][ 98. 128.]]

Theano

Theano

URL http://deeplearning.net/software/theano/index.html

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano is most commonly used to perform Deep Learning and has excellent GPUsupport and integration through PyCUDA. The following steps can be used to setup and configure Theano on yourown profile.

About Theano on ShARC


Theano is available on ShARC as both Singularity images and by local installation.


http://deeplearning.net/software/theano/index.html


As Theano and all its dependencies are written in Python. A Singularity image is available for instant usage and it canalso be installed locally in your home directory. For local installation, the use of Anaconda (Python) is recommendedas it is able to create a virtual environment in your home directory, allowing the installation of new Python packageswithout admin permission.


Theano Singularity Images



/usr/local/packages/singularity/images/theano/gpu.img


singularity exec --nv /usr/local/packages/singularity/images/theano/gpu.img /bin/bash


singularity exec --nv /usr/local/packages/singularity/images/theano/gpu.img python→˓your_theano_script.py




Theano is installed as part of Anaconda and can be found inside the image at:

/usr/local/anaconda3-4.2.0/lib/python3.5/site-packages/theano


Image Index



– /usr/local/packages/singularity/images/theano/gpu.img

• GPU Images

– Latest: 0.9.0-GPU-Ubuntu16.04-CUDA8-cudNN5.0-Anaconda3.4.2.0

* Path: /usr/local/packages/singularity/images/theano/0.9.0-GPU-Ubuntu16.04-CUDA8-cudNN5.0-Anaconda3.4.2.0.img

* Def file: /sharc/software/apps/singularity/theano.def

2.3. ShARC 299





Local Installation


Load the relevant modules with the following command:

module load apps/python/anaconda3-4.2.0module load libs/cudnn/5.1/binary-cuda-8.0.44

Create a conda environment to load relevant modules on your local user account:

conda create -n theano python=3.5source activate theano

Install the other Python module dependencies which are required using conda:

conda install -y numpy scipy nose pydot-ngconda install -y theano pygpu

For optimal Theano performance, enable the CUDA memory manager CNMeM. To do this, create the .theanorc file inyour HOME directory and set the fraction of GPU memory reserved by Theano. The exact amount of energy may haveto be hand-picked: if Theano asks for more memory that is currently available on the GPU, an error will be thrownduring import of theano module. Create or edit the .theanorc file with nano:

nano ~/.theanorc

Add the following lines and, if necessary, change the 0.8 number to whatever works for you

[lib]cnmem=0.8

Run python and verify that Theano is working correctly

python -c "import theano;theano.test()"


The previous instuctions installs Theano and its dependencies inside your home directory but every time you use a newsession or within your job scripts, the modules must be loaded and conda must be activated again. Use the followingcommand to activate the Conda environment with Theano installed:

module load apps/python/anaconda3-4.2.0module load libs/cudnn/5.1/binary-cuda-8.0.44source activate theano

Torch

Torch

URL http://torch.ch/


http://torch.ch/


Torch is a scientific computing framework with wide support for machine learning algorithms that puts GPUs first.It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDAimplementation.

About Torch on ShARC


Torch is available on ShARC as both Singularity images and as a module.


Torch Singularity Images



/usr/local/packages/singularity/images/torch/gpu.img


singularity exec --nv /usr/local/packages/singularity/images/torch/gpu.img /bin/bash


singularity exec --nv /usr/local/packages/singularity/images/torch/gpu.img th→˓yourscript.lua





Image Index



– /usr/local/packages/singularity/images/torch/gpu.img

• GPU Images

– Latest: v7-GPU-Ubuntu16.04-CUDA8-cudNN5.0 (GCC 5.4.0)

2.3. ShARC 301





* Path: /usr/local/packages/singularity/images/torch/v7-GPU-Ubuntu16.04-CUDA8-cudNN5.0.img

* Def file: /sharc/software/apps/singularity/torch_gpu.def

Using the Torch Module

The following is an instruction on how to use the Torch module.


Load the Torch module which also loads anaconda 3.4, CUDA 8.0, cuDNN 5.1 and GCC 4.9.4.

module load apps/torch/nvidia-7/gcc-4.9.4-cuda-8.0-cudnn-5.1-conda-3.4

On the DGX-1, load the NCCL library optimised for the hardware:

moudle libs/nccl/dgx-1/binary-cuda-8.0

On any other node use a generic NCCL build:

moudle load libs/nccl/generic/gcc-4.9.4-cuda-8.0

(Optional if you plan to interface with python) Create a conda environment to load relevant modules on your local useraccount and activate it

conda create -n torch python=3.5source activate torch


In the interactive session or your batch script, load the relevant modules and (optionally) activate your conda environ-ment

module load apps/torch/nvidia-7/gcc-4.9.4-cuda-8.0-cudnn-5.1-conda-3.4

#Optionalsource activate torch

TURBOMOLE

TURBOMOLE

Version 7.2

Dependencies No additional modules loaded.

URL http://www.turbomole.com/

Documentation http://www.turbomole-gmbh.com/turbomole-manuals.html

TURBOMOLE: Program Package for ab initio Electronic Structure Calculations.


http://www.turbomole.com/

http://www.turbomole-gmbh.com/turbomole-manuals.html


Usage

TURBOMOLE 7.2 can be activated using the module file:

module load apps/turbomole/7.2/binary

An example of a TURBOMOLE executable is jobex. Note: TURBOMOLE is configured on ShARC to use theserial and SMP executables (i.e. export PARA_ARCH=SMP).

Important note: Only licensed users of TURBOMOLE are entitled to use the code; refer to TURBOMOLE’swebsite for license details: http://www.turbomole-gmbh.com/how-to-buy.html. Access to TURBOMOLE onShARC is restricted to members of the unix group turbomole. To be added to this group, please [email protected] and provide evidence of your eligibility to use TURBOMOLE.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run ridft_smp and which is submitted to the queue by typing qsub my_job.sh.


module load apps/turbomole/7.2/binary

export TURBOTMPDIR=$TMPDIRexport PARNODES=4

ridft_smp > ridft_smp.out

The script requests 4 cores using the Open MP parallel environment smp with a runtime of 30 mins and 2 GB of realmemory per core. The TURBOMOLE input files are required to be in the directory where you run your job.

Installation notes

TURBOMOLE 7.2 was installed as a binary installation using the install_turbomole.sh script; the modulefile is /usr/local/modulefiles/apps/turbomole/7.2/binary.

The TURBOMOLE 7.2 installation was tested by running the TTEST command in the $TURBODIR/TURBOTESTdirectory as part of the installation procedure.

VASP

VASP

Version 5.4.1 (05Feb16)

Dependencies Fortran and C compilers, an implementation of MPI, numerical libraries BLAS, LA-PACK, ScaLAPACK, FFTW. Modules for Intel compiler 17.0.0, Open MPI 2.0.1 and Intel MKL2017.0 loaded.

2.3. ShARC 303

http://www.turbomole-gmbh.com/how-to-buy.html


URL https://www.vasp.at/

Documentation https://www.vasp.at/index.php/documentation

The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g.electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

Usage

VASP 5.4.1 can be activated using the module file:

module load apps/vasp/5.4.1.05Feb16/intel-17.0.0-openmpi-2.0.1

The VASP 5.4.1 executables are vasp_std, vasp_gam and vasp_ncl.

Important note: Only licensed users of VASP are entitled to use the code; refer to VASP’s website for license details:https://www.vasp.at/index.php/faqs. Access to VASP on ShARC is restricted to members of the unix group vasp. Tobe added to this group, please contact [email protected] and provide evidence of your eligibilityto use VASP.

Batch jobs

Users are encouraged to write their own batch submission scripts. The following is an example batch submissionscript, my_job.sh, to run vasp_std and which is submitted to the queue by typing qsub my_job.sh.


module load apps/vasp/5.4.1.05Feb16/intel-17.0.0-openmpi-2.0.1

mpirun vasp_std

The script requests 4 cores using the Open MPI parallel environment mpi with a runtime of 30 mins and 2 GB of realmemory per core.

Installation notes

VASP 5.4.1 (05Feb16) was installed using the install_vasp.sh script; the module file is /usr/local/modulefiles/apps/vasp/5.4.1.05Feb16/intel-17.0.0-openmpi-2.0.1.

The VASP 5.4.1 installation was tested by running a batch job using the my_job.sh batch script, above, and theinput for the “O atom” example (https://cms.mpi.univie.ac.at/wiki/index.php/O_atom) from the online VASP tutorials(https://cms.mpi.univie.ac.at/wiki/index.php/VASP_tutorials).

VCFtools


https://www.vasp.at/

https://www.vasp.at/index.php/documentation

https://www.vasp.at/index.php/faqs

https://cms.mpi.univie.ac.at/wiki/index.php/O_atom

https://cms.mpi.univie.ac.at/wiki/index.php/VASP_tutorials


VCFtools

Version 0.1.14

Dependancies gcc/5.4

URL https://vcftools.github.io/

Documentation https://vcftools.github.io/examples.html

VCFtools is a program package designed for working with VCF files, such as those generated by the 1000 GenomesProject. The aim of VCFtools is to provide easily accessible methods for working with complex genetic variation datain the form of VCF files.

Usage

VCFtools can be activated using the module file:

module load apps/vcftools/0.1.14

This will both add the binary files to the $PATH and the Perl module to $PERL5LIB.

Installation notes

VCFtools was compiled using the install_vcftools.sh script, the module file is 0.1.14.

Development Tools on ShARC

CMake

CMake is a build tool commonly used when compiling other libraries.

Usage

CMake can be loaded with:

module load dev/cmake/3.7.1/gcc-4.9.4

NB CMake has a run-time dependency on libstdc++ so the above also needs to (and does) load the GCC compilerversion 4.9.4.

Usage often involves:

1. Creating and cd-ing into a dedicated build directory within a source tree then

2. Running something like cmake -DSOME_OPTION -DANOTHER_OPTION ..

Installation

Version 3.7.1

1. Install using this script

2.3. ShARC 305

https://vcftools.github.io/

https://vcftools.github.io/examples.html


2. Install this modulefile as /usr/local/modulefiles/dev/cmake/3.7.1/gcc-4.9.4

Doxygen

Doxygen is a tool for building documentation for source code e.g. inter-related HTML pages for C++ source code.

Usage

Doxygen can be loaded with:

module load dev/doxygen/1.8.13/gcc-4.9.4

NB Doxygen has a run-time dependency on libstdc++ so the above also needs to (and does) load the GCC compilerversion 4.9.4.

Installation

Version 1.8.13

1. Install using this script

2. Install this modulefile as /usr/local/modulefiles/dev/doxygen/1.8.13/gcc-4.9.4

GNU Compiler Collection (gcc)

The GNU Compiler Collection (gcc) is a widely used, free collection of compilers including C (gcc), C++ (g++) andFortran (gfortran). The defaut version of gcc on the system is 4.8.5

$ gcc -v

Using built-in specs.COLLECT_GCC=gccCOLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapperTarget: x86_64-redhat-linuxConfigured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/→˓share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --→˓enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --→˓enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --→˓enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,→˓obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-→˓libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/→˓isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-→˓linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_→˓32=x86-64 --build=x86_64-redhat-linuxThread model: posixgcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)

It is possible to switch to other versions of the gcc compiler suite using modules. After connecting to ShARC (seeEstablishing a SSH connection), start an interactive sesssion with the qrshx command. Choose the version of thecompiler you wish to use using one of the following commands



module load dev/gcc/6.2module load dev/gcc/5.4

Confirm that you’ve loaded the version of gcc you wanted using gcc -v.

Documentation

man pages are available on the system. Once you have loaded the required version of gcc, type

man gcc



Installation Notes

These notes are primarily for system administrators:

Version 6.2

• Installation script

• gcc 6.2 modulefile saved as /usr/local/modulefiles/dev/gcc/6.2

Version 5.4

• Installation script

• this modulefile, saved as /usr/local/modulefiles/dev/gcc/5.4

Version 4.9.4

• this script

• this modulefile, saved as /usr/local/modulefiles/dev/gcc/4.9.4

Intel Compilers

Intel Compilers help create C, C++ and Fortran applications that can take full advantage of the advanced hardwarecapabilities available in Intel processors and co-processors. They also simplify that development by providing highlevel parallel models and built-in features like explicit vectorization and optimization reports.

Versions

The Intel compilers were installed as part of Intel Parallel Studio but can be used with or without the other ParallelStudio components.

2.3. ShARC 307




After connecting to the ShARC cluster (see Establishing a SSH connection), start an interactive session with theqrshx or qrsh command then activate a specific version of the compilers using one of:

module load dev/intel-compilers/17.0.0/binarymodule load dev/intel-compilers/16.0.1/binarymodule load dev/intel-compilers/15.0.7/binary


C

To compile the C hello world example into an executable called hello using the Intel C compiler:

icc hello.c -o hello

C++

To compile the C++ hello world example into an executable called hello using the Intel C++ compiler:

icpc hello.cpp -o hello

Fortran

To compile the Fortran hello world example into an executable called hello using the Intel Fortran compiler:

ifort hello.f90 -o hello


Once you have loaded the module on ShARC, man pages are available for Intel compiler products:

man ifortman icc

The following links are to Intel’s website:

• User and Reference Guide for the Intel® C++ Compiler 17.0

• User and Reference Guide for the Intel® Fortran Compiler 17.0

• Step by Step optimizing with Intel C++ Compiler



Related Software on the system

TODO: add link to NAG Fortran Library (serial)


https://software.intel.com/en-us/intel-cplusplus-compiler-17.0-user-and-reference-guide-intel-system-studio-2017

https://software.intel.com/en-us/intel-fortran-compiler-17.0-user-and-reference-guide

https://software.intel.com/en-us/articles/step-by-step-optimizing-with-intel-c-compiler


Installation Notes


Intel Compilers 17.0.0


This modulefile was installed as /usr/local/modulefiles/dev/intel-compilers/17.0.0.


Installed as part of Parallel Studio Composer Edition 2016.1.





Intel Parallel Studio

Intel Parallel Studio XE Composer Edition is a software development suite that helps boost application performanceby taking advantage of the ever-increasing processor core count and vector register width available in Intel Xeonprocessors and other compatible processors. Its core components are compilers and libraries for fast linear algebra,data transformation and parallelisation.

The suite includes:

• Intel C++ and Fortran compilers: these help create C, C++ and Fortran applications that “can take full advan-tage of the advanced hardware capabilities available in Intel processors and co-processors. They also simplifythat development by providing high level parallel models and built-in features like explicit vectorization andoptimization reports”.

• Data Analytics Acceleration Library (DAAL): functions for data analysis (characterization, summarization, andtransformation) and Machine Learning (regression, classification, and more). Only available in the 2017 versionof Parallel Studio.

• Math Kernel Library (MKL): This library provides highly optimized, threaded and vectorized functions to max-imize performance on each processor family. Utilizes de facto standard C and Fortran APIs for compatibilitywith BLAS, LAPACK and FFTW functions from other math libraries.

• Integrated Performance Primitives (IPP): “high-quality, production-ready, low-level building blocks for imageprocessing, signal processing, and data processing (data compression/decompression and cryptography) appli-cations.”

• Threading Building Blocks (TBB): lets you write “parallel C++ programs that take full advantage of multicoreperformance, that are portable and composable, and that have future-proof scalability.”

It does not include Intel’s MPI implementation. See Intel’s site for further details of what Parallel Studio ComposerEdition includes.

2.3. ShARC 309

https://software.intel.com/en-us/intel-parallel-studio-xe/details



Some components of Parallel Studio are freely available for those only wanting community support; other components,such as the compilers are commercial software.

The University has purchased a number of Intel-supported licenses for the Parallel Studio Composer Edition productslisted above. These can be accessed by (and are shared between) the University’s HPC clusters and researchers’ ownmachines.

Installation Notes



1. Download parallel_studio_xe_2017_composer_edition.tgz from the Intel Portal, save in /usr/local/media/protected/intel/2017.0/ then make it only readable by the app-adminsgroup.

2. Ensure details of the Intel license server are in the file /usr/local/packages/dev/intel-pe-xe-ce/license.lic

3. Run this script. This installs Parallel Studio into /usr/local/packages/dev/intel-pe-xe-ce/2017.0/binary/. Products are activated using the aforementioned license fileduring the installation process.


• The Compilers modulefile should be installed as /usr/local/modulefiles/dev/intel-compilers/17.0.0.

• The DAAL modulefile should be installed as /usr/local/modulefiles/libs/intel-daal/2017.0/binary.

• The IPP modulefile should be installed as /usr/local/modulefiles/libs/intel-ipp/2017.0/binary.

• The MKL modulefile should be installed as /usr/local/modulefiles/libs/intel-mkl/2017.0/binary.

• The TBB modulefile should be installed as /usr/local/modulefiles/libs/intel-tbb/2017.0/binary.




1. Download parallel_studio_xe_2016_composer_edition_update1.tar from the Intel Por-tal, save in /usr/local/media/protected/intel/2016.1/ then make it only readable by the





app-admins group.





• The DAAL modulefile should be installed as /usr/local/modulefiles/libs/intel-daal/2016.1/binary.







1. Download l_compxe_2015.7.235.tgz from the Intel Portal, save in /usr/local/media/protected/intel/2015.7/ then make it only readable by the app-admins group.







2.3. ShARC 311







NAG Fortran Compiler

The NAG Fortran Compiler is robust, highly tested, and valued by developers all over the globe for its checking capa-bilities and detailed error reporting. The Compiler is available on Unix platforms as well as for Microsoft Windowsand Apple Mac platforms. Release 6.0 has extensive support for both legacy and modern Fortran features, and alsosupports parallel programming with OpenMP.

Making the NAG Compiler available

After connecting to ShARC (see Establishing a SSH connection), start an interactive sesssion with the qrshx or qrshcommand. To make a version of the NAG Fortran Compiler available, run one of following module commands:

module load dev/NAG/6.1module load dev/NAG/6.0


To compile the Fortran hello world example into an executable called hello using the NAG compiler

nagfor hello.f90 -o hello


Once you’ve run one of the NAG Compiler module command, man documentation is available:

man nagfor

Online documentation:

• PDF version of the NAG Fortran Compiler Manual

• NAG Fortran Compiler Documentation Index (NAG’s Website)

• Differences between different versions can be found in the Release Notes.

Installation Notes

The following notes are primarily for system administrators:

Licensing

Add the license key(s) to /usr/local/packages/dev/NAG/license.lic.

The license key needs to be updated annually before 31st July.




http://www.nag.co.uk/nagware/np/r60_doc/np60_manual.pdf

http://www.nag.co.uk/nagware/np.asp

https://www.nag.co.uk/nag-compiler


The NAG compiler environment modules (see below) set the environment variable $NAG_KUSARI_FILE to the pathto this file.

Version 6.1

1. Perform an unattended install using this script. The software will be installed into /usr/local/packages/dev/NAG/6.1.

2. Install this modulefile as /usr/local/modulefiles/dev/NAG/6.1

3. Test the installation by compiling and building a sample Fortran 90 program

module load dev/NAG/6.1nagfor -o /tmp/f90_util /usr/local/packages/dev/NAG/6.1/lib/NAG_Fortran/f90_util.→˓f90/tmp/f90_util

Version 6.0

First, run the following

nag_vers=6.0media_dir="/usr/local/media/NAG/$nag_vers"mkdir -p $media_dirtarball=npl6a60na_amd64.tgztarball_url=https://www.nag.co.uk/downloads/impl/$tarball

wget -c $tarball_url -P $media_dir

mkdir -m 2775 -p $prefixchown -R $USER:app-admins $prefix

mkdir -p $tmp_dirpushd $tmp_dirtar -zxf $media_dir/$tarballpushd NAG_Fortran-amd64/

Next, run the interactive install script

./INSTALL.sh

Accept the license and answer the questions as follows:

• Install compiler binaries to where? [/usr/local/bin]? /usr/local/packages/dev/NAG/6.0/bin/

• Install compiler library files to where? /usr/local/packages/dev/NAG/6.0/lib/NAG_Fortran

• Install compiler man page to which directory? /usr/local/packages/dev/NAG/6.0/man/man1

• Suffix for compiler man page [1] leave as default

• Install module man pages to which directory? /usr/local/packages/dev/NAG/6.0/man/man3

• Suffix for module man pages [3]? leave as default

Install this modulefile as /usr/local/modulefiles/dev/NAG/6.0

Finally, test the installation by compiling and building a sample Fortran 90 program

module load dev/NAG/6.0nagfor -o /tmp/f90_util /usr/local/packages/dev/NAG/6.0/lib/NAG_Fortran/f90_util.f90/tmp/f90_util

2.3. ShARC 313


NVCC - Nvidia CUDA Compiler

To compile GPU code using the NVIDIA compiler, nvcc, first start an interactive session (see Interactive use of theGPUs). Next, you need to set up the compiler environment via one of the following module statements:

module load libs/cuda/8.0.44module load libs/cuda/7.5.18

depending on the version of CUDA you intend to use. This makes the nvcc CUDA compiler available.



• less than 5.0.0

It is therefore recommended that you load the most recent 4.x version of GCC when building CUDA programs onShARC:


An example of the use of nvcc:

nvcc filename.cu -arch sm_20

will compile the CUDA program contained in the file filename.cu. The -arch flag above signifies the computecapability of the intended GPU hardware, which is 2.0 for our current GPU modules. If you are intending to generatecode for older architectures you may have to specify sm_10 or sm_13 for example.

PGI Compilers

The PGI Compiler suite offers C,C++ and Fortran Compilers. For full details of the features of this compiler suite, seePGI’s website.

Making the PGI Compilers available

After connecting to the ShARC cluster (see Establishing a SSH connection), start an interactive session with the qrshxor qrsh command then activate a specific version of the compiler suite using one of:

module load dev/PGI-compilers/17.5module load dev/PGI-compilers/16.10

Once you’ve loaded the module, you can check the version with

pgcc --version


C

To compile a C hello world example into an executable called hello using the PGI C compiler


http://www.pgroup.com/products/pgiworkstation.htm


pgcc hello.c -o hello

C++

To compile a C++ hello world example into an executable called hello using the PGI C++ compiler

pgc++ hello.cpp -o hello

Fortran

To compile a Fortran hello world example into an executable called hello using the PGI Fortran compiler

pgf90 hello.f90 -o hello

Compiling on the GPU using the PGI Compiler

Start an interctive GPU session (Interactive use of the GPUs) and the following module command

module load dev/PGI-compilers/16.10

The PGI compilers have several features that make them interesting to users of GPU hardware:-

OpenACC Directives

OpenACC is a relatively new way of programming GPUs that can be significantly simpler to use than low-levellanguage extensions such as CUDA or OpenCL. From the OpenACC website :

The OpenACC Application Program Interface describes a collection of compiler directives to specifyloops and regions of code in standard C, C++ and Fortran to be offloaded from a host CPU to an attachedaccelerator. OpenACC is designed for portability across operating systems, host CPUs, and a wide rangeof accelerators, including APUs, GPUs, and many-core coprocessors.

The directives and programming model defined in the OpenACC API document allow programmers tocreate high-level host+accelerator programs without the need to explicitly initialize the accelerator, man-age data or program transfers between the host and accelerator, or initiate accelerator startup and shut-down.

For more details concerning OpenACC using the PGI compilers, see The PGI OpenACC website.

CUDA Fortran

In mid 2009, PGI and NVIDIA cooperated to develop CUDA Fortran. CUDA Fortran includes a Fortran 2003 compilerand tool chain for programming NVIDIA GPUs using Fortran.

• CUDA Fortran Programming Guide.

CUDA-x86

NVIDIA CUDA was developed to enable offloading computationally intensive kernels to massively parallel GPUs.Through API function calls and language extensions, CUDA gives developers explicit control over the mapping ofgeneral-purpose computational kernels to GPUs, as well as the placement and movement of data between an x86processor and the GPU.

2.3. ShARC 315

http://www.openacc-standard.org/About_OpenACC

http://www.pgroup.com/resources/accel.htm

http://www.pgroup.com/lit/whitepapers/pgicudaforug.pdf


The PGI CUDA C/C++ compiler for x86 platforms allows developers using CUDA to compile and optimize theirCUDA applications to run on x86-based workstations, servers and clusters with or without an NVIDIA GPU accelera-tor. When run on x86-based systems without a GPU, PGI CUDA C applications use multiple cores and the streamingSIMD (Single Instruction Multiple Data) capabilities of Intel and AMD CPUs for parallel execution.

• PGI CUDA-x86 guide.

Installation Notes

Version 17.5 The installer is interactive and the process was identical to that of version 16.10 below.

Version 16.10

The installer is interactive. Here is a log of the questions and answers.

A network installation will save disk space by having only one copy of thecompilers and most of the libraries for all compilers on the network, andthe main installation needs to be done once for all systems on the network.

1 Single system install2 Network install

Please choose install option: 1

Please specify the directory path under which the software will be installed.The default directory is /opt/pgi, but you may install anywhere you wish,assuming you have permission to do so.

Installation directory? [/opt/pgi] /usr/local/packages/dev/pgi

If you use the 2016 directory in your path, you may choose toupdate the links in that directory to point to the 16.10 directory.

Do you wish to update/create links in the 2016 directory? (y/n) yMaking symbolic links in /usr/local/packages/dev/pgi/linux86-64/2016

Installing PGI JAVA components into /usr/local/packages/dev/pgiaInstalling PGI CUDA components into /usr/local/packages/dev/pgiInstalling AMD GPU components into /usr/local/packages/dev/pgiInstalling PGI OpenACC Unified Memory components into /usr/local/packages/dev/pgi ...

************************************************************************MPI

************************************************************************This release contains version 1.10.2 of the Open MPI library.

Press enter to continue...

Do you want to install Open MPI onto your system? (y/n) yDo you want to enable NVIDIA GPU support in Open MPI? (y/n) y

Do you wish to generate license keys or configure license service? (y/n) nThe PGI license management script is available at:/usr/local/packages/dev/pgi/linux86-64/16.10/bin/pgi_license_tool

Do you want the files in the install directory to be read-only? (y/n) n


http://www.pgroup.com/resources/cuda-x86.htm


Modulefile

Version 17.5

The PGI compiler installer creates a suitable modulefile that’s configured to our system. It puts it at /usr/local/packages/dev/pgi/modulefiles/pgi64/17.5 so all that is required is to copy this to where we keep themodulefiles at /usr/local/modulefiles/dev/PGI-compilers/17.5

Version 16.10

The PGI compiler installer creates a suitable modulefile that’s configured to our system. It puts it at /usr/local/packages/dev/pgi/modulefiles/pgi64/16.10 so all that is required is to copy this to where we keepmodules at /usr/local/modulefiles/dev/PGI-compilers/16.10

sbt

sbt

Versions 0.13.13

URL http://www.scala-sbt.org/

sbt is a build tool for Scala, Java, and more.

Interactive Usage

After connecting to ShARC, start an interactive session with the qrshx command.

The latest version of sbt (currently 0.13.13) is made available with the command

module load dev/sbt


module load dev/sbt/0.13.13

This command makes the sbt command available to your session. You will also need to load a version of Java. Forexample


Installation notes


sbt 0.13.13 was installed as follows

2.3. ShARC 317

http://www.scala-sbt.org/


cp /usr/local/media/sbt/sbt-0.13.13.tgz .tar -xvzf ./sbt-0.13.13.tgzcd sbt-launcher-packaging-0.13.13/mkdir -p /usr/local/packages/dev/sbt/0.13.13mv ./* /usr/local/packages/dev/sbt/0.13.13

Installation Media

• The install tarball is on the system at /usr/local/media/sbt/sbt-0.13.13.tgz

Module files

• The module file is on the system at /usr/local/modulefiles/dev/sbt/0.13.13

• On github: 0.13.13.

Libraries on Sharc

FLAMEGPU

FLAME GPU

URL http://www.flamegpu.com/

FLAMEGPU is a multi agent simulation framework developed at the University of Sheffield. It uses GPUs to accel-erate the simulation of multiagent systems but abstracts the GPU architecture away from users so that they can writemodels using a high level of abstraction (without having to write GPU code).

Installing FLAMEGPU on ShARC

To install FLAME GPU you should checkout the latest master branch of the code which has Linux support

git clone https://github.com/FLAMEGPU/FLAMEGPU.git

To compile the FLAME GPU examples you will need to be on a GPU node. You can start an interactive session using

qrshx -l gpu=1 -P gpu -l rmem=15G

You will then need to load the relevant modules

module load libs/CUDA/8.0.44/binarymodule load dev/gcc/4.9.4

FLAMEGPU allows you to specify a CUDA_PATH environment variable so that you can change the CUDA versioneasily. To set this for the CUDA 8.0.44 module use

export CUDA_PATH=/usr/local/packages/libs/CUDA/8.0.44/binary/cuda

You can now navigate to the FLAME GPU examples folder and build the examples. e.g.


http://www.flamegpu.com/

http://www.flamegpu.com


cd FLAMEGPU/examplesmake

You can now run the console versions of the example models by navigating to FLAMEGPU/bin/x64 and calling theappropriate shell script. e.g.

cd ../..cd bin/x64./Boids_BruteForce_console.sh

FLAME GPU will run the example for one iteration and output the model state to a file 1.xml in the model’s iterationsdirectory.

Visualisation currently does not work with X forwarding as FLAME GPU uses complex rendering techniques whichare not supported. A solution using VirtualGL is in progress.

armadillo

armadillo


URL http://arma.sourceforge.net/

Armadillo is a high quality linear algebra library (matrix maths) for the C++ language, aiming towards a good balancebetween speed and ease of use.

Usage

To make this library available, run the following:

module load libs/armadillo/7.950.1/gcc-4.9.4

To compile this example program

#include <iostream>#include <armadillo>

using namespace std;using namespace arma;

int main(){mat A = randu<mat>(4,5);mat B = randu<mat>(4,5);

cout << A*B.t() << endl;

return 0;}

If you save the above program as example.cpp, once you load the module it can be compiled using

2.3. ShARC 319

http://arma.sourceforge.net/

http://arma.sourceforge.net/docs.html#example_prog


g++ example.cpp -o example -O2 -larmadillo

Installation notes


Version 7.950.1

This was compiled with GCC 4.9.4 and the Intel MKL

• Run this script

• Next, this modulefile as /usr/local/modulefiles/libs/armadillo/7.950.1/gcc-4.9.4

BLAS

The BLAS (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performingbasic vector and matrix operations. The Level 1 BLAS perform scalar, vector and vector-vector operations, the Level2 BLAS perform matrix-vector operations, and the Level 3 BLAS perform matrix-matrix operations. Because theBLAS are efficient, portable, and widely available, they are commonly used in the development of high quality linearalgebra software, LAPACK for example.

Usage

ShARC runs on the CentOS 7.x flavour of Linux which provides a standard build of BLAS via RPM files. We havemade this version of BLAS available via the following module command

$ module load libs/blas/3.4.2-5/binary

Installation notes


Version 3.4.2-5 binary

BLAS was extracted from the RPMs for blas and blas-devel

# Install script # Module File as /usr/local/modulefiles/modulefiles/libs/blas/3.4.2-5/binary

Boost C++ Library

Boost C++ Library


URL www.boost.org



Boost provides free, peer-reviewed and portable C++ source libraries.

Usage

To activate the library run:

module load libs/boost/1.64.0/gcc-4.9.4

Note that this also activates a particular version of the GCC compiler (as Boost depends on the C++ standardlibrary, which is provided by the compiler). You must use this version of the compiler to build your code if you wantto use this build of the Boost library. If you want to use a different compiler / compiler version then you need to requestthat a new build of Boost be compiled or compile a new build yourself.

Boost has been built without Python support: use conda and install the boost conda package if you want to use Boostwith Python.

Build a simple program using Boost

Many Boost libraries are header-only which makes them particularly simple to compile. The following program readsa sequence of integers from standard input, uses Boost.Lambda to multiply each number by three, and writes themto standard output (taken from http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html):

#include <boost/lambda/lambda.hpp>#include <iostream>#include <iterator>#include <algorithm>

int main(){

using namespace boost::lambda;typedef std::istream_iterator<int> in;

std::for_each(in(std::cin), in(), std::cout << (_1 * 3) << " " );

}

Copy this into a file called example1.cpp and compile with:

g++ example1.cpp -o example

Provided you loaded the correct modules given above, the program should compile without error.

Linking to a Boost library

The following program is taken from the official Boost documentation http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html

#include <boost/regex.hpp>#include <iostream>#include <string>

int main(){

std::string line;

2.3. ShARC 321





boost::regex pat( "^Subject: (Re: |Aw: )*(.*)" );

while (std::cin){

std::getline(std::cin, line);boost::smatch matches;if (boost::regex_match(line, matches, pat))

std::cout << matches[2] << std::endl;}

}

This program makes use of the Boost.Regex library, which has a separately-compiled binary component we needto link to. Assuming that the above program is called example2.cpp, compile with the following command:

g++ example2.cpp -o example2 -lboost_regex

If you get an error message that looks like this:

example2.cpp:1:27: error: boost/regex.hpp: No such file or directory

the most likely cause is that you forgot to load the correct modules as detailed above.

Installation Notes


Boost in turn was installed as a dependency of Caffe. It in turn depends on the libunistring and icu libraries.

Version 1.64.0

Built using:

• gcc 4.9.4

• icu 58.2

• libunistring 0.9.7

Has run-time dependencies on those three modules (gcc is required for the C++ standard library).

Built without Python support.

1. Download, configure, build, test and install using this script (installation log: install.log).

2. Install this modulefile as /usr/local//modulefiles/libs/boost/1.64.0/gcc-4.9.4

3. Test by compiling and running the two programs shown above.

CUDA

CUDA, which stands for Compute Unified Device Architecture, is a parallel computing platform and application pro-gramming interface (API) model created by NVIDIA. It allows software developers to use a CUDA-enabled graphicsprocessing unit (GPU) for general purpose processing - an approach known as General Purpose GPU (GPGPU).



Usage

There are several versions of the CUDA library available. As with many libraries installed on the system, CUDAlibraries are made available via module commands which are only available once you have started a qrshx sessionwith a GPU resource -l gpu=. For example, to request a single GPU:

qrshx -l gpu=1

See Using GPUs on ShARC for more information on how to request a GPU-enabled node for an interactivesession or job submission.

The latest version CUDA is loaded with the command:

module load libs/CUDA

Alternatively, you can load a specific version with one of the following:

module load libs/CUDA/8.0.44/binarymodule load libs/CUDA/7.5.18/binary

To check which version of CUDA you are using

$ nvcc --versionnvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2016 NVIDIA CorporationBuilt on Sun_Sep__4_22:14:01_CDT_2016Cuda compilation tools, release 8.0, V8.0.44



• less than 5.0.0

It is therefore recommended that you load the most recent 4.x version of GCC when building CUDA programs:

module load dev/gcc/4.9.4

Compiling the sample programs

You do not need to be using a GPU-enabled node to compile the sample programs but you do need a GPU to run them.

In a qrsh session

# Load modulesmodule load libs/CUDA/8.0.44/binarymodule load dev/gcc/4.9.4

# Copy CUDA samples to a local directory# It will create a directory called NVIDIA_CUDA-8.0_Samples/mkdir cuda_samplescd cuda_samplescp -r $CUDA_SDK .

# Compile (this will take a while)cd NVIDIA_CUDA-8.0_Samples/make

2.3. ShARC 323


A basic test is to run one of the resulting binaries, deviceQuery.

Documentation

• CUDA Toolkit Documentation

• The power of C++11 in CUDA 7

Profiling using nvprof

Note that nvprof, NVIDIA’s CUDA profiler cannot write output to the /fastdata filesystem.

This is because the profiler’s output is a SQLite <https://www.sqlite.org/>__ database and SQLite requires a filesystemthat supports file locking but file locking is not enabled on the (Lustre) filesystem mounted on /fastdata (forperformance reasons).

CUDA Training

GPUComputing@sheffield provides a self-paced introduction to CUDA training course.

Determining the NVIDIA Driver version

Run the command:

cat /proc/driver/nvidia/version

Example output is

NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.81 Wed Aug 17 22:24:07 PDT 2016GCC version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)

Installation notes


Device driver

The NVIDIA device driver is installed and configured using the gpu-nvidia-driver systemd service (managedby puppet). This service runs /usr/local/scripts/gpu-nvidia-driver.sh at boot time to:

• Check the device driver version and uninstall it then reinstall the target version if required;

• Load the nvidia kernel module;

• Create several device nodes in /dev/.

The NVIDIA device driver is currently version 384.81.


http://docs.nvidia.com/cuda/index.html#axzz3uLoSltnh

http://devblogs.nvidia.com/parallelforall/power-cpp11-cuda-7/

http://lustre.org/

http://gpucomputing.shef.ac.uk



CUDA 8.0.44

1. The CUDA toolkit binaries and samples were installed using a binary .run file:

cuda_vers="8.0.44"prefix="/usr/local/packages/libs/CUDA/${cuda_vers}/binary"mkdir -m 2775 -p $prefixchown ${USER}:app-admins $prefixcd /usr/local/media/nvidia/chmod +x cuda_${cuda_vers}_linux.run./cuda_${cuda_vers}_linux.run --toolkit --toolkitpath=${prefix}/cuda \

--samples --samplespath=${prefix}/samples \--no-opengl-libs -silent

2. This modulefile was installed as /usr/local/modulefiles/libs/CUDA/8.0.44/binary

CUDA 7.5.18

1. The CUDA toolkit binaries and samples were installed using a binary .run file as per version 8.0.44.

2. This modulefile was installed as /usr/local/modulefiles/libs/CUDA/7.5.18/binary

cuDNN

cuDNN

Dependencies CUDA, gcc

URL https://developer.nvidia.com/cudnn

Documentation https://developer.nvidia.com/cudnn

The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deepneural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backwardconvolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.

Usage

Only GPU-enabled nodes are able to run the library. See Using GPUs on ShARC for more information on howto request a GPU-enabled node for an interactive session or job submission.

Load the appropriate cuDNN version (and implicitly load a specific CUDA version) with one of the followingcommands:

module load libs/cudnn/6.0/binary-cuda-8.0.44module load libs/cudnn/5.1/binary-cuda-8.0.44module load libs/cudnn/5.1/binary-cuda-7.5.18

Installation notes


2.3. ShARC 325




The cuDNN library is only available to download through the developer portal. Installation .tgz files are located in/usr/local/media/protected/cudnn.

Version 6.0

• Install script: install_cudnn6.0_for_cuda8.0.sh

• Module file

Version 5.1

• Install script: install_cudnn5.1_for_cuda7.5_cuda8.0.sh

• Module file for CUDA 8.0

• Module file for CUDA 7.5

fftw

fftw


URL http://www.fftw.org/

FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, ofarbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sinetransforms or DCT/DST).

Usage

To make this library available, run the following:

module load libs/fftw/3.3.5/gcc-4.9.4

Installation notes


Version 3.3.5

This was compiled with GCC 4.9.4 (for compatibility with CUDA, which doesn’t support GCC >= 5.0.0). Thefollowing were enabled at compile-time:

• Threading (inc. OpenMP)

• Shared-library support

• SIMD (specifically, AVX2)



http://www.fftw.org/



During the testing stage you should see lots of numerical output plus:


--------------------------------------------------------------


--------------------------------------------------------------

Next, this modulefile as /usr/local/modulefiles/libs/fftw/3.3.5/gcc-4.9.4

Geospatial Data Abstraction Library (GDAL)

GDAL


URL http://www.gdal.org/

GDAL is a library used by many Geographic Information Systems (GIS) packages for converting between manydifferent raster and vector GIS data formats. It also includes command-line utilities for data translation and processing.It is released under an an X/MIT style Open Source license by the Open Source Geospatial Foundation.

Usage

By running

$ module load libs/gdal/2.2.0/gcc-6.2

you

• add several GDAL programs to your PATH environment variable

• allow other programs to make use of (dynamically link against) the GDAL library

• activate version 4.9.4 of the GCC compiler suite (as its C++ standard library is required by GDAL)

You can run gdal-config --version to test that you are running the required version

$ gdal-config --version2.2.0

The command-line programs that GDAL provides are:

• gdaladdo

• gdalbuildvrt

• gdal-config

• gdal_contour

• gdaldem

• gdalenhance

2.3. ShARC 327


http://www.osgeo.org/


• gdal_grid

• gdalinfo

• gdallocationinfo

• gdalmanage

• gdal_rasterize

• gdalserver

• gdalsrsinfo

• gdaltindex

• gdaltransform

• gdal_translate

• gdalwarp

• nearblack

• ogr2ogr

• ogrinfo

• ogrlineref

• ogrtindex

• testepsg

The gdal... utilities are mostly for processing raster GIS data formats, whilst the ogr... utilities are for process-ing vector GIS data formats.

Documentation

Standard man pages are available for the provided commands/functions.


$ man gdal_contour

Much more information is available on the project site.

Supported file formats

GDAL has been compiled on this system with support for only a limited set of GIS data formats. See Installationnotes below for a list of those provided by each available version of GDAL.

Installation notes


Version 2.2.0

GDAL 2.2.0 was compiled with v4.9.4 of the GCC compiler suite.

1. Download, configure, build and install by switching to a scratch directory and running this script, ensuringthat all stderr and stdout is redirected to a log file.




2. Install this modulefile as /usr/local/modulefiles/libs/gdal/2.2.0/gcc-4.9.4

The file formats supported by this build are listed in the compilation log file.

geos

geos

Version 3.6.1

URL http://trac.osgeo.org/geos/

GEOS (Geometry Engine, Open Source) is a C/C++ port of a subset of the Java Topology Suite (JTS), which in turnis a library that:

> provides an object model for planar geometry together with a set of fundamental geometric functions. > JTSconforms to the Simple Features Specification for SQL published by the Open GIS Consortium. > JTS is designed tobe used as a core component of vector-based geomatics software such as geographical information systems. > It canalso be used as a general-purpose library providing algorithms in computational geometry.

Usage

To make this library available, run the following module commands

module load libs/geos/3.6.1/gcc-4.9.4

This also activates version 4.9.4 of the GCC compiler suite (as its C++ standard library is required by GEOS)

The rgeos interface in R

rgeos is a CRAN package that provides an R interface to geos. It is not installed in R by default so you need to installa version in your home directory.

After connecting to ShARC (see Establishing a SSH connection), start an interactive session (e.g. using the qrshxcommand). Run the following module commands

module load apps/R/3.2.0module load compilers/gcc/4.8.2module load libs/gcc/4.8.2/geos/3.6.1

Launch R and run the command

install.packages('rgeos')

If you’ve never installed an R package before on the system, it will ask you if you want to install to a personal library.Answer y to any questions you are asked.

The library will be installed to a sub-directory called R in your home directory and you should only need to performthe above procedure once.

Once you have performed the installation, you will only need to run the module commands above to make the geoslibrary available to the system. Then, you use rgeos as you would any other library in R

2.3. ShARC 329

http://trac.osgeo.org/geos/

http://locationtech.github.io/jts/

http://www.opengeospatial.org/standards/sfs


library('rgeos')

Installation notes


Version 3.6.1

GEOS 3.6.1 was compiled with v4.9.4 of the GCC compiler suite.

1. Download, configure, build, test and install using this script, ensuring that all stderr and stdout is redi-rected to a log file.

2. Install this modulefile as /usr/local/modulefiles/libs/geos/3.6.1/gcc-4.9.4

Potentially useful output at the end of the configure run

Swig: falsePython bindings: falseRuby bindings: false

The build was tested by running make check from the build directory:

# TOTAL: 1# PASS: 1# SKIP: 0# XFAIL: 0# FAIL: 0# XPASS: 0

icu - International Components for Unicode

icu

Latest Version 58.2

Dependencies gcc

URL http://site.icu-project.org/

ICU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for softwareapplications. ICU is widely portable and gives applications the same results on all platforms and between C/C++ andJava software.

ICU is released under a nonrestrictive open source license that is suitable for use with both commercial software andwith other open source or free software.

Usage

Version 58.2 of the icu library for C requires gcc version 4.9.4 (for the C++ standard library); To make the library andthis compiler available, run the following:


http://site.icu-project.org/


module load libs/icu/58.2/gcc-4.9.4

Installation Notes


Version 58.2

This Icu 58.2 build links against the GCC 4.9.4 C++ standard library and was installed as a dependency of boost_sharc(build using the same C++ standard library); Boost in turn was installed as a dependency of Caffe.

1. Download, configure, build, test and install using this script

2. Check the console output of the install process for All tests OK

3. Install this modulefile as /usr/local/modulefiles/libs/icu/58.2/gcc-4.9.4

Intel Data Analytics Acceleration Library

Intel’s Data Analytics Acceleration Library (DAAL) provides functions for data analysis (characterization, summa-rization, and transformation) and machine learning (regression, classification, and more).


DAAL can be used with and without other Parallel Studio packages. To access it use one of:

module load libs/intel-daal/2017.0/binarymodule load libs/intel-daal/2016.1/binary



Installation Notes


Intel DAAL 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/intel-daal/2017.0/binary.

Intel DAAL 2016.1


This modulefile was installed as /usr/local/modulefiles/libs/intel-daal/2016.1/binary.

2.3. ShARC 331


Intel Integrated Performance Primitives

Integrated Performance Primitives (IPP) are “high-quality, production-ready, low-level building blocks for image pro-cessing, signal processing, and data processing (data compression/decompression and cryptography) applications.”


IPP can be used with and without other Parallel Studio packages. To access it, use one of the following:

module load libs/intel-ipp/2017.0/binarymodule load libs/intel-ipp/2016.1/binarymodule load libs/intel-ipp/2015.7/binary



Installation Notes


Intel IPP 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/intel-ipp/2017.0/binary.

Intel IPP 2016.1



Intel IPP 2015.7



Intel Math Kernel Library

Intel’s Math Kernel Library (MKL) provides highly optimized, threaded and vectorized functions to maximize per-formance on each processor family. It Utilises de-facto standard C and Fortran APIs for compatibility with BLAS,LAPACK and FFTW functions from other math libraries.


MKL can be used with and without other Parallel Studio packages. To access it run one of the following:

module load libs/intel-mkl/2017.0/binarymodule load libs/intel-mkl/2016.1/binarymodule load libs/intel-mkl/2015.7/binary



Sample C and Fortran programs demonstrating matrix multiplication are available for version 2017.0 in the directory$MKL_SAMPLES:

$ ls $MKL_SAMPLES/mkl_c_samples mkl_fortran_samples

To compile one of these you need to copy the samples to a directory you can write to, ensure the MKL and the Intelcompilers are both loaded, change into the directory containing the relevant sample program (C or Fortran) then runmake to compile:

$ qrsh$ cp -r $MKL_SAMPLES/ ~/mkl_samples$ module load dev/intel-compilers/17.0.0$ module load libs/intel-mkl/2017.0/binary$ cd ~/mkl_samples/mkl_fortran_samples/matrix_multiplication$ make

ifort -c src/dgemm_example.f -o release/dgemm_example.oifort release/dgemm_example.o -mkl -static-intel -o release/dgemm_exampleifort -c src/dgemm_with_timing.f -o release/dgemm_with_timing.oifort release/dgemm_with_timing.o -mkl -static-intel -o release/dgemm_with_timingifort -c src/matrix_multiplication.f -o release/matrix_multiplication.oifort release/matrix_multiplication.o -mkl -static-intel -o release/matrix_→˓multiplicationifort -c src/dgemm_threading_effect_example.f -o release/dgemm_threading_effect_→˓example.oifort release/dgemm_threading_effect_example.o -mkl -static-intel -o release/dgemm_→˓threading_effect_example

You should then find several compiled programs in the release directory:

$ ./release/matrix_multiplicationThis example measures performance of computing the realmatrix product C=alpha*A*B+beta*C usinga triple nested loop, where A, B, and C are matricesand alpha and beta are double precision scalars

Initializing data for matrix multiplication C=A*B formatrix A( 2000 x 200) and matrix B( 200 x 1000)

Intializing matrix data

Making the first run of matrix product usingtriple nested loop to get stable run timemeasurements

Measuring performance of matrix product usingtriple nested loop

== Matrix multiplication using triple nested loop ==== completed at 184.42246 milliseconds ==

Example completed.

For documentation and a tutorial, start an interactive session using qrshx, load MKL, then run:

$ firefox $MKL_SAMPLES/mkl_fortran_samples/matrix_multiplication/tutorial/en/index.htm

2.3. ShARC 333




Installation Notes


Intel MKL 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/intel-mkl/2017.0/binary.

Intel MKL 2016.1



Intel MKL 2015.7



Intel Threading Building Blocks

Threading Building Blocks (TBB) lets you write “parallel C++ programs that take full advantage of multicore perfor-mance, that are portable and composable, and that have future-proof scalability.”

Interactive usage

TBB can be used with and without other Parallel Studio packages.

To make use of it you first need to start an interactive session that has access to multiple cores. Here we request sixcores from the clusters’ scheduler:

$ qrsh -pe smp 6

Next, we need to active a specific version of TBB. Run one of the following:

$ module load libs/intel-tbb/2017.0/binary$ module load libs/intel-tbb/2016.1/binary$ module load libs/intel-tbb/2015.7/binary

You can find sample TBB programs for TBB 2017.0 in the directory $TBB_SAMPLES:

$ ls $TBB_SAMPLES

common concurrent_hash_map concurrent_priority_queue GettingStarted graph index.→˓html license.txt parallel_do parallel_for parallel_reduce pipeline task task_→˓arena task_group test_all



We can compile and run a sample program by copying the samples to our (writable) home directory, moving in to thedirectory containing the sample program (which should have a Makefile) then running make:

$ cp -r $TBB_SAMPLES ~/tbb_samples$ cd ~/tbb_samples/GettingStarted/sub_string_finder/$ ls

license.txt Makefile readme.html sub_string_finder.cpp sub_string_finder_extended.→˓cpp sub_string_finder_pretty.cpp

$ make

g++ -O2 -DNDEBUG -o sub_string_finder sub_string_finder.cpp -ltbb -lrtg++ -O2 -DNDEBUG -o sub_string_finder_pretty sub_string_finder_pretty.cpp -ltbb -lrtg++ -O2 -DNDEBUG -o sub_string_finder_extended sub_string_finder_extended.cpp -ltbb -→˓lrt./sub_string_finder_extendedDone building string.Done with serial version.Done with parallel version.Done validating results.

Serial version ran in 3.79692 secondsParallel version ran in 0.794959 secondsResulting in a speedup of 4.77625

Many of the sample directories contain HTML documentation. To read this you need to start an interactive graphicalsession (using qrshx or qrshx) then run:

$ firefox ~/tbb_samples/index.html

Tutorials

See Chrys Woods’ excellent tutorials for C++ programmers and Python programmers.



Installation Notes


Intel TBB 2017.0


This modulefile was installed as /usr/local/modulefiles/libs/intel-tbb/2017.0/binary.

2.3. ShARC 335

http://chryswoods.com/parallel_c++

http://chryswoods.com/parallel_python/index.html


Intel TBB 2016.1



Intel TBB 2015.7



LAPACK

LAPACK is written in Fortran 90 and provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. The associatedmatrix factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also provided, as are related computa-tions such as reordering of the Schur factorizations and estimating condition numbers. Dense and banded matrices arehandled, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices,in both single and double precision.

Usage

ShARC runs on the CentOS 7.x flavour of Linux which provides a standard build of LAPACK via RPM files. We havemade this version of LAPACK available via the following module command

$ module load libs/lapack/3.4.2-5/binary

Installation notes


Version 3.4.2-5 binary

LAPACK was extracted from the RPMs for lapack and lapack-devel

# Install script # Module File as /usr/local/modulefiles/modulefiles/libs/lapack/3.4.2-5/binary

libunistring

libunistring


URL http://www.gnu.org/software/libunistring/

Text files are nowadays usually encoded in Unicode, and may consist of very different scripts - from Latin letters toChinese Hanzi, with many kinds of special characters: accents, right-to-left writing marks, hyphens, Roman numbers,and much more. But the POSIX platform APIs for text do not contain adequate functions for dealing with particular


http://www.gnu.org/software/libunistring/


properties of many Unicode characters. In fact, the POSIX APIs for text have several assumptions at their base whichdon’t hold for Unicode text.

This library provides functions for manipulating Unicode strings and for manipulating C strings according to theUnicode standard.

Usage

To make the library available, run the following:

module load libs/libunistring/0.9.7/gcc-4.9.4

This correctly populates the environment variables LD_LIBRARY_PATH, LIBRARY_PATH and CPATH.

Installation Notes


Version 0.9.7

This build was installed as a dependency of boost_sharc (build using the same C++ standard library); Boost in turnwas installed as a dependency of Caffe.

1. Download, configure, build, test and install using this script

2. Check the console output of the install process to check that no tests have errored/failed: TOTAL: 499 /PASS: 489 / SKIP: 10

3. Install this modulefile as /usr/local//modulefiles/libs/libunistring/0.9.7/gcc-4.9.4

OpenSSL

OpenSSL

Latest Version provided by the operating system

URL https://www.openssl.org/

> OpenSSL is an open source project that provides a robust, commercial-grade, and full-featured toolkit for the Trans-port Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library.

Usage

If user applications are installed in a central location by a system administrator then they are typically made avail-able using one or more Environment Modules to allow users to select the version(s) they want to use, facilitatingreproducible research.

However, OpenSSL has been installed as an operating system package on ShARC so:

• There is only one version available;

2.3. ShARC 337

https://www.openssl.org/


• There is no need to load an Environment Module to use it.

There are two reasons for this:

• It is more important to use the latest version of OpenSSL (for security reasons) than it is to continue to provideolder versions for reproducible research purposes;

• Relatively few new features are added between OpenSSL versions so concerns about reproducible research aremarginal.

To check the version of OpenSSL that has been installed using the operating system’s package manager run:

openssl version

Conda

If you are using the conda package manager (e.g. if using Anaconda/Miniconda) to maintain software environments(e.g. for Python or R) in your home directory then you will find that conda installs its own openssl package.

Note that this conda package is not managed by system administrators: if you install it (which you will do wheneveryou create a new conda environment) then you should be aware that it will not be automatically updated if securityvulnerabilities are found and a new version released. However, you can update it yourself periodically using:

conda upgrade openssl

or if you are using R:

conda upgrade -c r openssl r-openssl

Installation Notes

The openssl RPM is installed by default in Centos 7.x. The openssl-devel RPM (header files) was installedusing Puppet.

PROJ.4

PROJ.4


URL https://github.com/OSGeo/proj.4

PROJ.4 consists of programs and a library for managing cartographic projections.

Usage

By running

$ module load libs/proj/4.9.3/gcc-4.9.4

you


https://github.com/OSGeo/proj.4


• add several PROJ.4 programs to your PATH environment variable

• allow other programs to make use of (dynamically link against) the PROJ.4 library

• activate version 4.9.4 of the GCC compiler (as its C++ standard library is required when using PROJ.4)

You can run proj to test that you are running the required version

$ projRel. 4.9.3, 15 August 2016usage: proj [ -bCeEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]

Documentation

Standard man pages are available for the provided commands/functions cs2cs, geod, proj, geodesic andpj_init.


$ man proj

Installation notes


Version 4.9.3

PROJ.4 4.9.3 was compiled with v4.9.4 of the GCC compiler suite.


2. Download, build and install using this script, ensuring that all stderr and stdout is redirected to a logfile.

3. Install this modulefile as /usr/local/modulefiles/libs/proj/4.9.3/gcc-4.9.4

Test by running make check from the build directory. For this build this yielded:

============================================Running ../nad/test27 using ../src/proj:============================================doing tests into file proj_out27, please waitdiff proj_out27 with pj_out27.distTEST OKtest file proj_out27 removed

../nad/test83 ../src/proj============================================Running ../nad/test83 using ../src/proj:============================================doing tests into file proj_out83, please waitdiff proj_out83 with pj_out83.distTEST OKtest file proj_out83 removed

PROJ_LIB=. ../nad/testvarious ../src/cs2csUsing locale with comma as decimal separator============================================

2.3. ShARC 339


Running ../nad/testvarious using ../src/cs2cs:============================================doing tests into file tv_out, please waitRel. 4.9.3, 15 August 2016<lt-cs2cs>: while processing file: <stdin>, line 1pj_transform(): invalid x or yRel. 4.9.3, 15 August 2016<lt-cs2cs>: while processing file: <stdin>, line 2pj_transform(): acos/asin: |arg| >1.+1e-14diff tv_out with tv_out.distTEST OKtest file tv_out removed

SCOTCH

SCOTCH

Version 6.0.4

Dependancies GCC compiler and Open MPI

URL http://scotch.gforge.inria.fr/

Documentation https://gforge.inria.fr/docman/?group_id=248

Software package and libraries for sequential and parallel graph partitioning, static mapping and clustering, sequentialmesh and hypergraph partitioning, and sequential and parallel sparse matrix block ordering

Usage

SCOTCH and PT SCOTCH 6.0.4 can be activated using the module file:

module load libs/scotch/6.0.4/gcc-6.2-openmpi-2.0.1

Installation notes

SCOTCH was installed using the install_scotch.sh script; the module file is gcc-6.2-openmpi-2.0.1.

The installation was tested using the ‘’make check” command as part of running the above installation script.

Parallel Systems

MVAPICH2 (Intel version)

MVAPICH2 (Intel version)

Latest Version 2.3b

Dependancies Intel compilers


http://scotch.gforge.inria.fr/

https://gforge.inria.fr/docman/?group_id=248


URL http://mvapich.cse.ohio-state.edu/

MVAPICH, also known as MVAPICH2, is, like OpenMPI, an implementation of the MPI standard for passing mes-sages between machines in parallel computing environments.

Versions

You can load MVAPICH2 using

module load mpi/mvapich2/2.3b/intel-17.0.0

This version was build with the Intel compiler suite version 17.0.0.

See here for an overview of the features offered per MVAPICH2 release.

Running MPI programs

A set of benchmarking programs programs, the OSU Micro Benchmarks, are available in the directory:

${MPI_HOME}/libexec/osu-micro-benchmarks/mpi/

Documentation and source code for these benchmarks can be found online.

To run one of these programs or another program built with this particular MPI software you want to submit anMPI-aware batch job (or interactive session) in which you load the MVAPICH2 module file then run the program.

An example batch job that runs a benchmark program using 4 CPU cores:

#!/bin/bash#$ -pe mpi 4

# Display the scheduler's allocation of 'slots' to the job# (where slots are equivalent to CPU cores in this case)echo $PE_HOSTFILE

# Load an MPI implementationmodule load mpi/mvapich2/2.3b/intel-17.0.0

# run an benchmark program on all of your MPI nodes using the mpirun utilitympirun $MVAPICH_ENV $MPI_HOME/libexec/osu-micro-benchmarks/mpi/collective/osu_→˓allgather

# or to run on a subset of the nodes:# (here we run a program that benchmarks bidirectional bandwidth between just 2 MPI→˓'ranks' (processes))mpirun -np 2 $MVAPICH_ENV $MPI_HOME/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bibw

Note: You should always specify the absolute or relative path to the MPI program you want to run; otherwise youwill see errors like:

[proxy:0:[email protected]] HYDU_create_process (utils/launch/launch.c:75):→˓execvp error on file mpi_hello_world (No such file or directory)[proxy:0:[email protected]] HYDU_create_process (utils/launch/launch.c:75):→˓execvp error on file mpi_hello_world (No such file or directory)

2.3. ShARC 341

http://mvapich.cse.ohio-state.edu/

http://mvapich.cse.ohio-state.edu/overview/

http://mvapich.cse.ohio-state.edu/benchmarks/


Building applications

To build an MPI program (e.g. mpi_hello_world.c):

# Connect to ShARCssh user@sharc

# Start an interactive sessionqrsh

# Load an MPI implementationmodule load mpi/mvapich2/2.3b/intel-17.0.0

cd path/to/source/code

# Compile the program using MVAPICH2's MPI-aware wrapper for the Intel 17.0 C compilermpicc mpi_hello_world.c

Installation notes


Version 2.3b, Intel 17.0.0 compiler

1. Ensure Intel compilers 17.0.0 are installed and licensed.

2. Download, compile and install MVAPICH2 2.3b using install.sh. The console log from runninginstall.sh.

3. Install this modulefile as /usr/local/modulefiles/mpi/mvapich2/2.3b/intel-17.0.0

4. Tested by running the OSU micro benchmarks using a mvapich2test.sge job submission script. Results:mvapich2test.sge.log.

5. NB the MVAPICH_ENV environment variable is set by the module file and is used to pass multiple environmentvariables to slaves for controlling process binding. See the module file for more info.




Dependancies gcc


The Open MPI Project is an open source Message Passing Interface implementation that is developed and maintainedby a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise,




technologies, and resources from all across the High Performance Computing community in order to build the bestMPI library available. Open MPI offers advantages for system and software vendors, application developers andcomputer science researchers.

Versions

You can load a specific version using

module load mpi/openmpi/2.1.1/gcc-6.2module load mpi/openmpi/2.1.1/gcc-5.4module load mpi/openmpi/2.1.1/gcc-4.9.4module load mpi/openmpi/2.0.1/gcc-6.2module load mpi/openmpi/2.0.1/gcc-5.4module load mpi/openmpi/2.0.1/gcc-4.9.4module load mpi/openmpi/1.10.4/gcc-6.2module load mpi/openmpi/1.10.4/gcc-4.9.4

See here for a brief guide to the new features in OpenMPI 2.x and here for a detailed view of the changes betweenOpenMPI versions.

CUDA: Note that if you are using CUDA with OpenMPI then you currently need to use a version of CUDA built withGCC < 5.0. C++ bindings If you are using the C++ bindings then you should use OpenMPI 1.10.4 as the bindingshave been deprecated in OpenMPI 2.0.1.

Examples

Example programs are available in the $MPI_HOME/examples/ directory.

To compile and run these programs: copy that directory to your home directory, start an interactive MPI-aware sessionon a worker node, activate the version of OpenMPI you wish to use, compile the examples then run them.

In more detail


# Start an interactive session from which we can run MPI processes using a core on→˓each of four nodesqrsh -pe mpi 4

# Load an MPI implementationmodule load mpi/openmpi/2.1.1/gcc-6.2

# Copy the examples to your home directorycp -r $MPI_HOME/examples ~/openmpi_2.1.1_examples

# Compile all programs in the examples directorycd ~/openmpi_2.1.1_examplesmake

# Once compiled, run an example program on all (or a subset) of your MPI nodes using→˓the mpirun utilitympirun -np 4 hello_c

Hello, world, I am 0 of 4, (Open MPI v2.1.1, package: Open MPI user@sharc-node003.→˓shef.ac.uk Distribution, ident: 2.1.1, repo rev: v2.1.0-100-ga2fdb5b, May 10, 2017,→˓141)

2.3. ShARC 343

https://mail-archive.com/[email protected]/msg00085.html

https://raw.githubusercontent.com/open-mpi/ompi/v2.x/NEWS


Hello, world, I am 1 of 4, (Open MPI v2.1.1, package: Open MPI user@sharc-node003.→˓shef.ac.uk Distribution, ident: 2.1.1, repo rev: v2.1.0-100-ga2fdb5b, May 10, 2017,→˓141)Hello, world, I am 2 of 4, (Open MPI v2.1.1, package: Open MPI user@sharc-node003.→˓shef.ac.uk Distribution, ident: 2.1.1, repo rev: v2.1.0-100-ga2fdb5b, May 10, 2017,→˓141)Hello, world, I am 3 of 4, (Open MPI v2.1.1, package: Open MPI user@sharc-node003.→˓shef.ac.uk Distribution, ident: 2.1.1, repo rev: v2.1.0-100-ga2fdb5b, May 10, 2017,→˓141)

Installation notes


Version 2.1.1, gcc 6.2

1. Enable GCC 6.2.0.

2. Download, compile and install OpenMPI 2.1.1 using the this script.

3. Install this modulefile as /usr/local/modulefiles/mpi/openmpi/2.1.1/gcc-6.2





Version 2.1.1, gcc 4.8.5


2. Install this modulefile as /usr/local/modulefiles/mpi/openmpi/2.1.1/gcc-4.8.5






1. Enable GCC 5.4.0








2. Download, compile and install OpenMPI 1.10.4 using this script.



3. Configure the OpenMPI Modular Component Architecture (MCA) by copying this script and save as/usr/local/packages/mpi/openmpi/1.10.4/gcc-6.2/openmpi-mca-params.conf; thisconfigures:

• the mtl (MCA Matching Transport Layer) to use the psm2 driver (i.e. use the high-bandwidth, low-latency Intel OmniPath fabric);

• the btl (MCA Byte Transport Layer) to use Omnipath but (not not Ethernet);

• the oob (MCA out of band messaging) to use the intra-cluster Ethernet fabric (specified using a networkaddress in CIDR format rather than by specifying Ethernet interface name, which can vary between nodes).





3. Configure the OpenMPI Modular Component Architecture (MCA) by copying this script to /usr/local/packages/mpi/openmpi/1.10.4/gcc-4.9.4//openmpi-mca-params.conf; thisconfigures:





OpenMPI (Intel version)

OpenMPI (intel version)


Dependancies Intel compilers



Versions


module load mpi/openmpi/2.0.1/intel-17.0.0module load mpi/openmpi/2.0.1/intel-15.0.7module load mpi/openmpi/1.10.4/intel-15.0.7

2.3. ShARC 345




C++ bindings If you are using the C++ bindings then you should use OpenMPI 1.10.4 as the bindings have beendeprecated in OpenMPI 2.0.1.

Examples

Example programs are available in the /usr/local/packages/mpi/openmpi/XXX/intel-17.0.0/examples/ directory (where XXX is the version of OpenMPI you are using e.g. 2.0.1).


In more detail



# Load an MPI implementationmodule load mpi/openmpi/2.0.1/intel-17.0.0




Hello, world, I am 0 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)Hello, world, I am 1 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)Hello, world, I am 2 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)Hello, world, I am 3 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)

Installation notes






Version 2.0.1, Intel 17.0.0 compiler



3. Install this modulefile as /usr/local/modulefiles/mpi/openmpi/2.0.1/intel-17.0.0

4. Test by running some Examples.









3. Configure the OpenMPI Modular Component Architecture (MCA) by copying this script to /usr/local/packages/mpi/openmpi/1.10.4/intel-15.0.7/openmpi-mca-params.conf; thisconfigures:






OpenMPI (pgi version)



Dependancies PGI Compiler


2.3. ShARC 347




Versions


module load mpi/openmpi/2.0.1/pgi-17.5

This also loads the relevant version of the PGI compiler (17.5 in this case).


Examples

Example programs are available in the $MPI_HOME/examples/ directory.


In more detail



# Load an MPI implementationmodule load mpi/openmpi/2.0.1/pgi-17.5




Hello, world, I am 0 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)Hello, world, I am 1 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)Hello, world, I am 2 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)





Hello, world, I am 3 of 4, (Open MPI v2.0.1, package: Open MPI user@sharc-node002.→˓shef.ac.uk Distribution, ident: 2.0.1, repo rev: v2.0.0-257-gee86e07, Sep 02, 2016,→˓141)

Installation notes


Version 2.0.1, PGI 17.5

1. Download, compile and install OpenMPI 2.0.1 using this script

2. Install this modulefile as /usr/local/modulefiles/mpi/openmpi/2.0.1/pgi-17.5

ShARC and Iceberg comparison

CPUs

The hardware on Iceberg is older and usually slower than that on ShARC. In particular, all of the CPUs on ShARCare ‘4th generation core’ Intel Haswell compared to a combination of ‘3rd generation core’ Intel Ivy Bridge, ‘1stgeneration core’ Intel Westmere and AMD Processors on Iceberg.

Most of the time, you can expect programs to run more quickly on ShARC than on Iceberg. How much more quicklydepends on which type of processor you were using on Iceberg and the characteristics of your program. One of thenew features of Intel Haswell CPUs is Fused Multiply-Add which can substantially increase the speed of floatingpoint calculations such as Matrix-Matrix Multiplications. Only programs that have been compiled to make use of thisfeature, such as R (Intel Build), will see a speed increase as a result.

It is theoretically possible for some programs to run more slowly on ShARC since the typical clock speed of a ShARCCPU is 2.4Ghz compared to 2.6Ghz on the Ivy Bridge nodes on Iceberg. You are advised to conduct tests to see whichsystem is best for your work. We are interested in any reproducible benchmarks you may run, please open an issue onour GitHub page to discuss this.

MPI Interconnect

Nodes on ShARC are connected together using Intel Omnipath which should be faster than the interconnect used onIceberg. We are interested in any reproducible benchmarks you may run, please open an issue on our GitHub page todiscuss this.

Off campus access

If you are off-campus, you will need to use a VPN connection to access ShARC. Iceberg can be accessed without aVPN Connection.

Software and Module names

The two systems have different suites of software installed and modules follow different naming schemes. See Soft-ware on ShARC and Software on iceberg for details of what’s installed on each system and how to access it.

2.3. ShARC 349

https://en.wikipedia.org/wiki/Haswell_(microarchitecture)

https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)

https://en.wikipedia.org/wiki/Westmere_(microarchitecture)

https://en.wikipedia.org/wiki/List_of_AMD_microprocessors

https://en.wikipedia.org/wiki/FMA_instruction_set

https://en.wikipedia.org/wiki/Matrix_multiplication



http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-architecture-fabric-overview.html


ShARC specifications

Total capacity

• Worker nodes: 121

• CPU cores: 2024

• Total memory: 12160 GiB

• GPUs: 40

• Fast network filesystem (Lustre): 669 TiB (/home and /data are shared with Iceberg)

Note that some of these resources have been purchased by research groups who have exclusive access to them.

General CPU node specifications

98 nodes are publicly available (not exclusive to research groups).

• Machine: Dell PowerEdge C6320

• CPUs: 2 x Intel Xeon E5-2630 v3

– Haswell processor microarchitecture;

– 2.40 GHz;

– Support for AVX2 vectorisation instructions (simultaneously apply the same operation to multiple valuesin hardware);

– Support for Fused Multiply-Add instructions (expedites operations involving the accummulation of prod-ucts e.g. matrix multiplication).

– Hyperthreading is disabled on all nodes bar four that are reserved for interactive jobs.

• RAM: 64 GB (i.e. 4 GiB / core)

– 1866 MHz;

– DDR4.

• Local storage: 1 TiB SATA III HDD

– /scratch: 836 GiB of temporary storage;

– /tmp: 16 GiB of temporary storage.

Large memory node specifications

Four nodes are publicly available (not exclusive to research groups).

These are identical to the general CPU nodes but with 256 GiB RAM (16 GiB per core).

GPU node specifications

Two nodes are publicly available (not exclusive to research groups):

• Machine: Dell PowerEdge C4130

• CPUs: 2 x Intel Xeon E5-2630 v3 (2.40GHz)

• RAM: 64 GB (i.e. 4 GiB / core); 1866 MHz; DDR4


http://lustre.org/

http://www.dell.com/uk/business/p/poweredge-c6320/pd

http://ark.intel.com/products/83356/Intel-Xeon-Processor-E5-2630-v3-20M-Cache-2_40-GHz

https://en.wikipedia.org/wiki/Haswell_(microarchitecture)

https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Advanced_Vector_Extensions_2

https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add

https://en.wikipedia.org/wiki/Hyper-threading

http://www.dell.com/uk/business/p/poweredge-c4130/pd


• Local storage: 800 GiB SATA SSD

• GPUs: 8 x NVIDIA Tesla K80

– 24 GiB of GDDR5 memory (12 GiB per GPU; 192 GiB per node)

– Up to 2.91 Teraflops of double precision performance with NVIDIA GPU Boost

– Up to 8.74 Teraflops of single precision performance with NVIDIA GPU Boost

Hardware-accellerated visualisation nodes

One node is publicly available:

• Machine: Dell Precision Rack 7910

• CPUs: 2 x Intel Xeon E5-2630 v3 (2.40GHz)

• RAM: 128 GiB (i.e. 8 GiB / core); 1866 MHz; DDR4

• Local storage: 1 TiB

• Graphics cards: 2x Quadro K4200

– Memory: 4 GiB GDDR5 SDRAM

Networking

• Intel OmniPath Architecture (OPA) (100 Gb/s) to all public nodes

• Gigabit Ethernet

Operating System and software

• OS: Centos 7.x (binary compatible with RedHat Enterprise Linux 7.x) on all nodes

• Interactive and batch job scheduling software: Son of Grid Engine

• Many applications, compilers, libraries and parallel processing tools. See Software on ShARC

Non-worker nodes

• Two login nodes (for resilience)

• Other nodes to provide:

– Lustre parallel filesystem

– Son of Grid Engine scheduler ‘head’ nodes

– Directory services

Using GPUs on ShARC



2.3. ShARC 351

http://www.nvidia.com/object/tesla-servers.html

http://www.dell.com/uk/business/p/precision-r7910-workstation/pd?oc=cu000pr7910mufws_




Interactive use of the GPUs

Once you are included in the GPU project group you may start using the GPU enabled nodes interactively by typing:

qrshx -l gpu=1

the -l gpu= parameter determines how many GPUs you are requesting. Currently, the maximum number of GPUsallowed per job is set to 8, i.e. you cannot exceed -l gpu=8. Most jobs will only make use of one GPU.

Interactive sessions provide you with 2 Gigabytes of CPU RAM by default which is significantly less than the amountof GPU RAM available. This can lead to issues where your session has insufficient CPU RAM to transfer data to andfrom the GPU. As such, it is recommended that you request enough CPU memory to communicate properly with theGPU

#Nvidia K80 GPU has 24GB of RAMqrshx -l gpu=1 -l rmem=25G

The above will give you 1GB more CPU RAM than the 24GB of GPU RAM available on the Nvidia K80.

Submitting batch GPU jobs

To run batch jobs on gpu nodes, edit your jobfile to include a request for GPUs, e.g. for a single GPU

#!/bin/bash#$ -l gpu=1

Requesting GPUs and multiple CPU cores from the scheduler

You may want to request multiple CPU cores on a single node with a certain number of GPUs per CPU core. Here ishow to request four CPU cores on a node with two GPU per CPU core:

#!/bin/bash#$ -pe smp 4#$ -l gpu=2

It is not currently possible to request:

• more CPU cores than GPUs

– e.g. a heterogeneous application which could use all 40 CPU cores in a given node whilst using all 8GPUs;

• non-multiple ratios of GPUs to CPUs

– e.g. a heterogeneous application which uses 4 GPUs, with 1 CPU core per GPU and can additionally useCPU cores for asynchronous host work i.e. an extra 16 cores, totalling 20 CPUs.

However, such scheduler requests may be supported in future.





ShARC GPU Resources

Hardware

Pulicly available GPU nodes

• GPU node specifications

Research group-specific GPU nodes:

• Nvidia DGX-1 [COM]

GPU-enabled Software

• Applications

– Caffe

– MATLAB

– Theano

– Tensorflow

– Torch

• Libraries

– CUDA

– cuDNN

• Development Tools

– PGI Compilers

– NVCC - Nvidia CUDA Compiler

Training materials

• Introduction to CUDA by GPUComputing@Sheffield

• Introducting to Deep Learning using Caffe on ShARC’s DGX-1 by GPUComputing@Sheffield

Deep Learning on ShARC

The availability of large amount of data, exponential power in parallel computation hardware and new algorightmshas sparked a revolution in the field of Neural Networks. In particular, the Deep Learning approach that uses NeuralNetwork with many layers allowing it to learn highly abstract concepts have been successfully applied to imageidenfification and segmentation, voice recognition, autonomous driving and machine translation.

Deep Learning Consultancy, Training & Support

The Research Software Engineering Sheffield (RSES) group is responsible for supporting Deep Learning software onShARC. Please file any Deep Learning related issues on our GPU Computing Github respository.

If you have a research project that requires Deep Learning expertise or support, members of the RSES teamare available to be costed on your grants. Contact [email protected] for more information.

2.3. ShARC 353




https://github.com/RSE-Sheffield/GPUComputing



Available Software

The following are the list of currently supported Deep Learning frameworks available on ShARC:

• Caffe

• Tensorflow

• Theano

• Torch

Deep Learning framworks often have a complex set of software requirements. With the introduction of Singularityon ShARC, a containerisation technology similar to Docker, it is now possible for you to create a software stack thatexactly fits your needs and will run on both your local development machine and the ShARC cluster (see Singularity).

Use of GPUs for training Neural Networks

The ability for GPUs to perform massive amounts of floating point operations and matrix multiplications in parallelmakes the hardware ideal for use in training Neural Network models and can often accelerate the process by at leastan order of magnitude compared to the use of standard CPUs.

See ShARC GPU Resources for more information on the GPU resources available on ShARC.

Nvidia DGX-1 Deep Learning Supercomputer

The Nvidia DGX-1 is the world’s first Deep Learning supercomputer. It is available to use for research groups withinthe Computer Science deparment. See Nvidia DGX-1 [COM] for more information.

Training Materials

• Introducting to Deep Learning using Caffe on ShARC’s DGX-1 by GPUComputing@Sheffield

Research Group-Specific Nodes on ShARC

Big memory nodes [COM]

The Department of Computer Science have purchased three nodes in ShARC that each have much more than thestandard amount of RAM per node.

Specifications

• 2x Intel Xeon E5-2630 v3 processors (2.40 GHz, 8 cores per processor i.e. 16 total)

• 768 GB RAM (48 GB / CPU core)

Requesting Access

The nodes are managed by the RSE group and are available by request to all research groups belonging to the ComputerScience Department.

To use the nodes you must join the rse Grid Engine (scheduler) Access Control List (ACL i.e. user group) and submitjobs using the rse Grid Engine Project.





To join this ACL please email RSE enquiries.

Requesting a node

Once you have obtained permission to use the nodes you can request an interactive session on one of the nodes using:

qrshx -P rse

Here -P rse specifies that you want to use the rse project for your session, which gives you access to thesebig memory nodes and ensures that your interactive session runs in the rse.q job queue (as can be seen if yousubsequently run qstat -u $USER from within your session).

Submitting jobs

Jobs can be submitted to the nodes by adding the -P rse parameter. For example, create a job script namedmy_job_script.sh with the contents:

#!/bin/bash#$ -P rse

echo "Hello world"

You can of course add more options to the script such as a request for additional RAM (e.g. ‘‘ $# -l rmem=10G‘‘).

Run your script with the qsub command

qsub my_job_script.sh

You can use the qstat command to check the status of your current job. An output file is created in your homedirectory that captures your script’s outputs.

See Starting interactive jobs and submitting batch jobs for more information on job submission and the Sun GridEngine scheduler.

Nvidia DGX-1 [COM]

The Nvidia DGX-1 is the world’s first Deep Learning supercomputer. It is equipped with 8 Tesla P100 GPUs con-nected together with NVLink technology to provide super fast inter-GPU communication. Capable of performing 170Teraflops of computation, it can provide up to 75 times speed up on training deep neural networks compared to thelatest Intel Xeon CPU.

DGX-1 Specifications

• 8x Tesla P100 GPUs (16GB RAM each)

• Dual 20-core Intel Xeon E5-2698 v4 2.2 GHz

• 512 GB System RAM

2.3. ShARC 355



Requesting Access

The node managed by the RSE group and is available by request to all research groups belonging to the ComputerScience Department.

To use the DGX-1 you must join the computer science RSE group and submit jobs to the RSE queue. To join thisgroup please email Twin Karmakharm or RSE enquiries.

Requesting a DGX-1 node

Once you have obtained permission to use the node, to request an interactive DGX-1 node, type:

qrshx -l gpu=1 -P rse -q rse.q

You will then be placed in a special RSE queue that has the DGX-1. The parameter -l gpu=1 determines the numberof GPUs that will be used in the job (maximum of 8), -P rse -q rse.q then denotes that you run the job underthe rse project with the rse.q queue. Note that the parameter -l gpu= is required otherwise you will be placedon a different node without a GPU (big memory node).

Submitting jobs

Jobs can be submitted to the DGX-1 by adding the -l gpu=1 -P rse -q rse.q parameters. For example,create a job script named my_job_script.sh with the contents:

#!/bin/bash#$ -l gpu=1 -P rse -q rse.q

echo "Hello world"

In the second line of the script #$ -l gpu=1 you can add more options such as memory request -l rmem=10Gor to be put on the DGX-1 queue -P rse -q rse.q.

Run your script with the qsub command

qsub my_job_script.sh

You can use qstat command to check the status of your current job. An output file is created in your home directorythat captures your script’s outputs.

See Starting interactive jobs and submitting batch jobs for more information on job submission and the Sun GridEngine scheduler.

Deep Learning on the DGX-1

Many popular Deep Learning packages are available to use on the DGX-1 and the ShARC cluster. Please see DeepLearning on ShARC for more information.

• Software on ShARC

• ShARC and Iceberg comparison

• ShARC specifications

• Using GPUs on ShARC

• Deep Learning on ShARC






• Research Group-Specific Nodes on ShARC

Parallel Computing

Modern computers contain more than one processor with a typical laptop usually containing either 2 or 4 (if you thinkyour laptop contains 8, it is because you have been fooled by Hyperthreading). Computer clusters such as ShARC orIceberg contain many hundreds of processors and the key to making your research faster is to learn how to distributeyour work across them. If your program is designed to run on only one processor, running it on our systems withoutmodification will not make it any faster (it may even be slower!). Learning how to use parallelisation technologies isvital.

This section explains how to use the most common parallelisation technologies on our systems.

If you need advice on how to parallelise your workflow, please contact the Research Software Engineering Group

SGE Array Jobs

The simplest way of exploiting parallelism on the clusters is to use Array Jobs.

An Array Job is a set of tasks run from a single batch job submission script. For example:

#!/bin/bash##$ -t 1-100#echo "Task id is $SGE_TASK_ID"

./myprog.exe $SGE_TASK_ID > output.$SGE_TASK_ID

The above script will submit 100 tasks to the job queues at once. The difference between each of these 100 jobsis the value of the environment variable $SGE_TASK_ID which will range from 1 to 100, determined by the line#$ -t 1-100. In this example, the program myprog.exe will be run 100 times with differing input values of$SGE_TASK_ID. 100 output files will be created with filenames output.1, output.2 and so on.

Advantages of Array Jobs:

• You only need to submit one job to run a series of very similar tasks;

• These tasks are independent and do not all need to run at once so the job scheduler can efficiently run one ormore queued tasks as the requested computational resources become available;

• They are particularly useful for Embarrassingly Parallel problems such as:

– Monte Carlo simulations (where $SGE_TASK_ID might correspond to random number seed);

– Parameter sensitivity analysis;

– Batch file processing (where $SGE_TASK_ID might refer to a file in a list of files to be processed).

Array Jobs on ShARC and Iceberg can have a maximum of 75000 tasks.

Limiting number of concurrent tasks for GPU array jobs

The job scheduler has means for ensuring fair use of the cluster if a user submits an array comprised of many CPU-only tasks. However, if a user submits an array of GPU jobs then this could result in unfair usage as at present the jobscheduler has mechanisms for lowering the priorities of pending tasks based on the user’s recent CPU and memoryusage but does not penalise for recent GPU usage.

2.4. Parallel Computing 357

https://en.wikipedia.org/wiki/Hyper-threading

http://rse.shef.ac.uk/contact/

https://en.wikipedia.org/wiki/Embarrassingly_parallel


Therefore, if you want to submit GPU array jobs you may want (or be asked) to explicitly limit the number of GPUtasks that can run concurrently.

Use the flag -tc 5 to specify the maximum number of concurrent tasks (five tasks in this example). As per the -tflag, it can either be included on the command line:

#Running 100 jobs with maximum of 5 running concurrentlyqsub -t 1-100 -tc 5 my_array_job.sh

or in the batch file itself:

#$ -t 1-100#$ -tc 5

Examples

• MATLAB SGE Job Array example

Managing output and error files

By default, when you run a Job Array a separate output and error file will be written per task to the directory theJob Array was submitted from. This may not be convenient: you may not want to be generating tens, hundreds orthousands of log files in your project’s directories on ShARC/Iceberg.

A more sensible approach could be to tell the scheduler to write output and error log files per task to a folder in yourhome directory called logs, also ensuring that the names of these files contain the job name:

#!/bin/bash##$ -N MyProjectX#$ -t 1-100#$ -o /home/$USER/logs#$ -e /home/$USER/logs#echo "Task id is $SGE_TASK_ID"

./myprog $SGE_TASK_ID

This will generate output files of the form:

MyProjectX.o$JOB_ID.$SGE_TASK_ID

and error files of the form:

MyProjectX.e$JOB_ID.$SGE_TASK_ID

where $SGE_TASK_ID is the task ID and $JOB_ID is the ID of the entire Array Job.

Grouping tasks for efficiency

If you know that each of your workflow’s tasks takes only a few seconds to run then making each of these a separatetask in your Job Array may not be particularly efficient as it takes a few seconds to schedule each job.

A better approach may be to batch up tasks into groups that you know will run for a few minutes each. This can beachieved by specifying a step size for the range of task IDs and then doing multiple things per task.


https://github.com/mikecroucher/HPC_Examples/tree/master/languages/MATLAB/SGE_array


In the example below, the submitted Array Job that consists of ten tasks which have SGE_TASK_ID values of 1, 11,21...91 (and therefore a step size of 10). This step size only needs to be specified once; later on in the script it can bedetermined using the $SGE_TASK_STEPSIZE variable. In the example, myprog is run ten times per task:

#!/bin/bash##$ -N MyProjectX#$ -t 1-100:10#$ -o /home/$USER/logs/#$ -e /home/$USER/logs/

echo "Startinng task $SGE_TASK_ID"for subtask_id in $(seq $SGE_TASK_ID $(( $SGE_TASK_ID + $SGE_TASK_STEPSIZE - 1 )) );→˓do

echo "Subtask $subtask_id of task $SGE_TASK_ID"./myprog $subtask_id

done

Email notifications

If you enable email notifications in your batch job submission script then you will receive emails for every task in yourArray Job. This helps you determine if any tasks in the Array Job failed but doesn’t help you determine if the entireArray Job has finished running. Here’s a sensible approach to managing email notifications:

1. Edit your Array Job submission script so you are only notified of aborted (a) tasks i.e.

#$ -M [email protected]#$ -m a

2. Then submit your Array Job like so:

[te1st@sharc-login1 ~]$ qsub my_array_job.sgeYour job-array 2035587.1-3:1 ("my_array_job.sge") has been submitted

3. Next, submit a very simple job that will only run when Array Job 2035587 has completed and emails youwhen it finishes:

[te1st@sharc-login1 ~]$ qsub -o /dev/null -e /dev/null -M [email protected] -m→˓ea -b y -l h_rt=00:00:15 -hold_jid 2035587 -N 'Array_Job_finished' trueYour job 2035588 ("Job_array_finished") has been submitted

You will therefore receive:

• An email for every failed task in the Array Job;

• An email shortly after the entire Array Job finishes.

Shared Memory Parallelism (SMP)

Shared-Memory Parallelism (SMP) is when work is divided between multiple threads or processes running on a singlemachine and these threads have access to common (shared) memory.

SMP is most performant when each thread / process has its own CPU core, which limits you to 16 processor coreson ShARC’s and (most of) Iceberg’s nodes. If you wish to scale to more parallel strands of execution or want morememory per strand than SMP can facilitate, you should consider more complex parallelisation schemes such as MPIor Hybrid OpenMP/MPI.


https://en.wikipedia.org/wiki/Thread_(computing)

https://en.wikipedia.org/wiki/Process_(computing)


OpenMP

OpenMP (Open Multi-Processing) is one of the most common programming frameworks used to implement shared-memory parallelism. It greatly simplifies the tasks of distributing work and coordination in a multi-threaded C, C++ orFortran program. Many research applications make use of the OpenMP library: you can often enable OpenMP supportat compile-time and/or run-time using compiler flags, command-line options and/or configuration files.

Batch OpenMP jobs

To submit a SMP job to our system, we we need to request a particular Parallel Environment in our batch jobsubmission scripts. On ShARC your script needs to include a line like:

#$ -pe smp N

whereas on Iceberg it needs to be:

#$ -pe openmp N

Change N to the number of processor cores you wish to use.

Here’s an example that requests 4 cores on ShARC:

#!/bin/bash# Request 4 cores from ShARC's scheduler#$ -pe smp 4#$ Request 4 GB of memory per CPU core#$ -l rmem=4G

# Tell programs that use the OpenMP library to use 4 coresexport OMP_NUM_THREADS=4./myprogram

Note that you have to specify the number of cores at least twice when running OpenMP jobs:

• Once to the scheduler (#$ -pe smp 4) so that it knows how many cores to assign to your job;

• Once to the OpenMP runtime environment (export OMP_NUM_THREADS=4) so that OpenMP knows howmany threads to create.

• You may also need to expliclty need to tell your application how many threads it should create.

Running other SMP schemes

Any system that uses shared memory parallelism, such as pthreads, Python’s multiprocessing module, R’s mcapplyand so on can be submitted using the smp (ShARC) / openmp (Iceberg) Parallel Environments. Many of these alsocheck the OMP_NUM_THREADS environment variable to determine how many threads to create.

Interactive SMP jobs

You can start an interactive SMP job on ShARC like so:

qrsh -pe smp 4 -l rmem=3G

Or like so on Iceberg:


http://openmp.org/wp/

https://en.wikipedia.org/wiki/POSIX_Threads

https://docs.python.org/3/library/multiprocessing.html

https://rforge.net/doc/packages/multicore/mclapply.html


qrsh -pe openmp 4 -l rmem=3G

Note: It may be difficult to start an interactive SMP session if the cluster(s) are busy.

Message Passing Interface (MPI)

The Message Passing Interface is a standard for passing data and other messages between running processes whichmay or may not be on a single computer. It is commonly used on computer clusters as a means by which a set of relatedprocesses can work together in parallel on one or more tasks. Unlike the SMP / OpenMP approaches to parallelism, theparallel strands of execution in a MPI environment do not share any memory: these strands (processes) must thereforecommunicate data and other information by passing messages between each other.

MPI is used on systems ranging from a few interconnected Raspberry Pi’s through to the UK’s national supercomputer,Archer.

MPI Implementations

The Message Passing Interface (MPI) itself is just a specification for a message passing library.

There are multiple implementations of this specification, each produced by a different organisation, including Open-MPI <https://www.open-mpi.org/> and MVAPICH. This documentation includes information on the MPI implementa-tions available on ShARC and Iceberg. These implementations have been compiled on our clusters in a way that allowsthem to make optimal use of each cluster’s high-speed network infrastructure (OmniPath on ShARC, Infiniband onIceberg). If you are not sure which implementation to use then try the latest available version of OpenMPI.

Batch MPI

To use MPI you need to:

• Include information in your batch job submission script that tells the Grid Engine scheduler you want to use aparticular Parallel Environment (mpi on ShARC, openmpi-ib on Iceberg);

• Use module load to activate a particular MPI implementation (or module load an application that itselfloads an MPI implementation behind the scenes).

Here is an example that requests 4 slots (CPU cores) with 8GB of RAM per slot then runs a program calledexecutable in the current directory using the OpenMPI library (version 2.0.1, built using version 6.2 of the gcccompiler). It is assumed that executable was previously compiled using that exact same MPI library. The ParallelEnvironment is specified using -pe.

#!/bin/bash# Request 4 MPI 'slots' (cores)#$ -pe mpi 4# Request 8GB of RAM per slot#$ -l rmem=8G

# Load a MPI librarymodule load mpi/openmpi/1.10.4/gcc-6.2

# Run a program previously compiled using that specific MPI librarympirun ./executable


https://en.wikipedia.org/wiki/Process_(computing)

http://thenewstack.io/installing-mpi-python-raspberry-pi-cluster-runs-docker/

http://www.archer.ac.uk/

http://mpi-forum.org/

http://mvapich.cse.ohio-state.edu/


Example MPI jobs

Some more example MPI jobs are available in the HPC Examples repository of Sheffield’s Research Software Engi-neering group

Interactive MPI

Our general-access interactive queues currently don’t have any MPI-compatible parallel environments enabled. Thus,it is not possible to run MPI jobs interactively.

MPI Training

Course notes from the national supercomputing centre are available here

Hybrid SMP / MPI

Overview

If you want to distribute work between a large number of cores (e.g. >16) on ShARC or Iceberg then the most commonapproach is MPI, as described here. The default MPI setup on ShARC and Iceberg does not ensure that your MPI jobis assigned the same number of CPU cores per node. However in certain cases you may want a symmetric distributionof of cores: you may want eight MPI processes, each of which internally parallelises work using four OpenMP threads,which would run optimally if each process were assigned four cores on a node (possibly running multiple processeson a node).

Usage on Iceberg

Support for Hybrid SMP/MPI is in the preliminary stages. Here is an example job submission for an executable thatcombines SMP (specifically OpenMP) with MPI:

#$ -pe openmpi-hybrid-4 8#$ -l rmem=2Gmodule load mpi/intel/openmpiexport OMP_NUM_THREADS=4mpirun -bynode -np 2 -cpus-per-proc 4 [executable + options]

There would be 2 MPI processes running, each with 4x 2GB of real memory shared between the four threads per MPIprocess, for a total of 8 x 2GB of real memory for the entire job. When we tried this, we got warnings saying that the-cpus-per-proc was getting deprecated. A quick google suggests that

mpirun -np 2 --map-by node:pe=1 [executable + options]

would be the appropriate replacement.

Here, -pe openmpi-hybrid-4 8 means ‘request eight slots (CPU cores) from the scheduler using theopenmpi-hybrid-4 Parallel Environment‘. This Parallel Environment ensure that the slots are assigned to nodesin groups of four. with this example, the job could be assigned four cores on each of two nodes or eight cores on onenode (given that all nodes have at least eight cores)

To see other available hybrid SMP/MPI Parallel Environments run qconf -spl | grep hybrid e.g.:


https://github.com/mikecroucher/HPC_Examples/tree/master/MPI



http://www.archer.ac.uk/training/course-material/2016/07/MPP_MPI_epcc/index.php


openmpi-hybrid-12openmpi-hybrid-16openmpi-hybrid-4openmpi-hybrid-6openmpi-hybrid-8

Note that the number of cores you request should be wholely divisible by the number in the name of the ParallelEnvironment you use.

Usage on ShARC

This is similar to Iceberg but at present only one Parallel Environnment is defined: mpi-smp-16. This particularParallel Environment can be used to request exclusive access to multiple nodes (as all standard nodes in ShARC have16 cores) e.g. including the following in your job submission script would result in your job being allocated threenodes:

#$ -pe mpi-smp-16 48

More Parallel Environments for Hybrid SMP/MPI can be added on request.

GPU Computing

Graphics Processing Units (GPUs) were, as the name suggests, originally designed for the efficient processing ofgraphics. Over time, they were developed into systems that were capable of performing general purpose computingwhich is why some people refer to modern GPUs as GP-GPUs (General Purpose Graphical Processing Units).

Graphics processing tpically involves relatively simple computations that need to applied to millions of on-screenpixels in parallel. As such, GPUs tend to be very quick and efficient at computing certain types of parallel workloads.

The GPUComputing@Sheffield website aims to facilitate the use of GPU computing within University of Sheffieldresearch by providing resources for training and purchasing of equipment as well as providing a network of GPU usersand research projects within the University.

GPU Community and NVIDIA Research Centre

The University of Sheffield has been officially affiliated with NVIDIA since 2011 as an NVIDIA CUDA ResearchCentre. As such NVIDIA offer us some benefits as a research institution including discounts on hardware, technicalliaisons, online training and priority seed hardware for new GPU architectures. For first access to hardware, trainingand details of upcoming events, discussions and help please join the GPUComputing google group.







https://research.nvidia.com/

https://developer.nvidia.com/academia/centers/university-sheffield-cuda-research-center

https://developer.nvidia.com/academia/centers/university-sheffield-cuda-research-center

https://groups.google.com/a/sheffield.ac.uk/forum/#!forum/gpucomputing




Using GPUs on Iceberg and ShARC

Hardware, sofware and ways to access GPUs are different on Iceberg and ShARC. Instructions specific to each clustercan be found below:

• Using GPUs on Iceberg

• Using GPUs on ShARC

Troubleshooting

In this section, we’ll discuss some tips for solving problems with ShARC and Iceberg. It is suggested that you workthrough some of the ideas here before contacting the CiCS helpdesk for assistance.

Frequently Asked Questions

I’m a new user and my password is not recognised

When you get a username on the system, the first thing you need to do is to syncronise your passwords which will setyour password to be the same as your network password.

Strange things are happening with my terminal

Symptoms include many of the commands not working and just bash-4.1$ being displayed instead of your user-name at the bash prompt.

This may be because you’ve deleted your .bashrc and .bash_profile files - these are ‘hidden’ files which livein your home directory and are used to correctly set up your environment. If you hit this problem you can run thecommand resetenv which will restore the default files.

I can no longer log onto iceberg

If you are confident that you have no password or remote-access-client related issues but you still can not log ontoiceberg you may be having problems due to exceeding your iceberg filestore quota. If you exceed your filestore quotain your /home area it is sometimes possible that crucial system files in your home directory gets truncated that effectthe subsequent login process.

If this happens, you should immediately email [email protected] and ask to be unfrozen.

I can not log into iceberg via the applications portal

Most of the time such problems arise due to due to Java version issues. As Java updates are released regularly, theseproblems are usually caused by the changes to the Java plug-in for the browser. Follow the trouble-shooting link fromthe iceberg browser-access page to resolve these problems. There is also a link on that page to test the functionalityof your java plug-in. It can also help to try a different browser to see if it makes any difference. All failing, you mayhave to fall back to one of the non-browser access methods.


https://www.shef.ac.uk/cics/password

http://www.sheffield.ac.uk/cics/research/hpc/using/access/browser



I cannot see my folders in /data or /shared

Some directories such as /data/<your username> or /shared/<your project/ are made available on-demand:. For example, if your username is ab1def and you look in /data straight after logging in, you may notsee /data/ab1def. The directory is there, it has just not been made available (via a process called mounting) to youautomatically. When you attempt to do something with the directory such as ls /data/ab1def or cd /data/ab1def, the directory will be mounted automatically and will appear to you. The directory will be automaticallyunmounted after a period of inactivity.

My batch job terminates without any messages or warnings

When a batch job that is initiated by using the qsub command or runfluent, runansys or runabaqus com-mands, it gets allocated specific amount of real memory and run-time. If a job exceeds either the real memory or timelimits it gets terminated immediately and usually without any warning messages.

It is therefore important to estimate the amount of memory and time that is needed to run your job to completion andspecify it at the time of submitting the job to the batch queue.

Please refer to the section on hitting-limits and estimating-resources for information on how to avoid these problems.

Exceeding your disk space quota

Each user of the system has a fixed amount of disk space available in their home directory. If you exceed this quota,various problems can emerge such as an inability to launch applications and run jobs. To see if you have exceededyour disk space quota, run the quota command:

quota

Size Used Avail Use% Mounted on10.1G 5.1G 0 100% /home/foo11b100G 0 100G 0% /data/foo11b

In the above, you can see that the quota was set to 10.1 gigabytes and all of this is in use. Any jobs submitted by thisuser will likely result in an Eqw status. The recommended action is for the user to delete enough files to allow normalwork to continue.

Sometimes, it is not possible to log in to the system because of a full quota, in which case you need to [email protected] and ask to the unfrozen.

I am getting warning messages and warning emails from my batch jobs about insufficient memory

If a job exceeds its real memory resource it gets terminated. You can use the rmem= parameter to increase the amountof real memory that your job requests.

What are the rmem (real memory) and (deprecated) mem (virtual memory) options?

Warning: The following is most likely only of interest when revisiting job submission scripts and documentationcreated before 26 June 2017 as now users only need to request real memory (rmem) and jobs are only killed if theyexceed their rmem quota (whereas prior to that date jobs could request and be policed using virtual memory memrequests).

2.5. Troubleshooting 365

https://www.shef.ac.uk/cics/research/hpc/iceberg/requirements


Running a program always involves loading the program instructions and also its data (i.e. all variables and arrays thatit uses) into the computer’s memory. A program’s entire instructions and its entire data, along with any dynamically-linked libraries it may use, defines the virtual storage requirements of that program. If we did not have cleveroperating systems we would need as much physical memory (RAM) as the virtual storage requirements of that pro-gram. However, operating systems are clever enough to deal with situations where we have insufficient real memory(physical memory, typically called RAM) to load all the program instructions and data into the available RAM. Thistechnique works because hardly any program needs to access all its instructions and its data simultaneously. There-fore the operating system loads into RAM only those bits (pages) of the instructions and data that are needed by theprogram at a given instance. This is called paging and it involves copying bits of the programs instructions and datato/from hard-disk to RAM as they are needed.

If the real memory (i.e. RAM) allocated to a job is much smaller than the entire memory requirements of a job ( i.e.virtual memory) then there will be excessive need for paging that will slow the execution of the program considerablydue to the relatively slow speeds of transferring information to/from the disk into RAM.

On the other hand if the RAM allocated to a job is larger than the virtual memory requirement of that job then it willresult in waste of RAM resources which will be idle duration of that job.

• The virtual memory limit defined by the -l mem cluster scheduler parameter defines the maximum amount ofvirtual memory your job will be allowed to use. This option is now deprecated - you can continue to submitjobs requesting virtual memory, however the scheduler no longer applies any limits to this resource.

• The real memory limit is defined by the -l rmem cluster scheduler parameter and defines the amount of RAMthat will be allocated to your job. The job scheduler will terminate jobs which exceed their real memory resourcerequest.

Note: As mentioned above, jobs now need to just request real memory and are policed using real memory usage. Thereasons for this are:

• For best performance it is preferable to request as much real memory as the virtual memory storage requirementsof a program as paging impacts on performance and memory is (relatively) cheap.

• Real memory is more tangible to newer users.

Insufficient memory in an interactive session

By default, an interactive session provides you with 2 Gigabytes of RAM (sometimes called real memory). You canrequest more than this when running your qrshx/qsh/qrsh command e.g.:

qrshx -l rmem=8G

This asks for 8 Gigabytes of RAM (real memory). Note that you should:

• not specify more than 256 GB of RAM (real memory) (rmem)

‘Illegal Instruction’ errors

If your program fails with an Illegal Instruction error then it may have been compiled using (and optimised for) onetype of processor but is running on another.

If you get this error after copying compiled programs onto Iceberg then you may need to recompile them on Icebergor recompile them elsewhere without agressively optimising for processor architecture.

If however you get this error when running programs on Iceberg that you have also compiled on the cluster thenyou may have compiled on one processor type and be running on a different type. You may not consistently get theillegal instruction error here as the scheduler may allocate you a different type of processor every time you run your



program. you can either recompile your program without optimisations for processor architecture or force your job torun on the type of processor it was compiled on using the -l arch= qsub/qrsh/qsh parameter e.g.

• -l arch=intel* to avoid being allocated one of the few AMD-powered nodes

• -l arch=intel-x5650 to use the Intel Westmere CPU architecture

• -l arch=intel-e5-26[567]0 to use the Intel Sandy Bridge CPU architecture

If you know the node that a program was compiled on but do not know the CPU architecture of that node then you candiscover it using the following command (substituting in the relevant node name):

qhost | egrep '(ARCH|node116)'

Windows-style line endings

If you prepare text files such as your job submission script on a Windows machine, you may find that they do not workas intended on the system. A very common example is when a job immediately goes into Eqw status after you havesubmitted it and you are presented with an error message containing:

failed searching requested shell because:

The reason for this behaviour is that Windows and Unix machines have different conventions for specifying ‘end ofline’ in text files. Windows uses the control characters for ‘carriage return’ followed by ‘linefeed’, \r\n, whereasUnix uses just ‘linefeed’ \n.

The practical upshot of this is that a script prepared in Windows using Notepad looking like this:

#!/bin/bashecho 'hello world'

will look like the following to programs on a Unix system:

#!/bin/bash\recho 'hello world'\r

If you suspect that this is affecting your jobs, run the following command on the system:

dos2unix your_files_filename

error: no DISPLAY variable found with interactive job

If you receive the error message:

error: no DISPLAY variable found with interactive job

the most likely cause is that you forgot the -X switch when you logged into iceberg. That is, you might have typed:

ssh [email protected]

instead of:

ssh -X [email protected]

macOS users might also encounter this issue if their XQuartz is not up to date.

2.5. Troubleshooting 367



Problems connecting with WinSCP

Some users have reported issues while connetcing to the system using WinSCP, usually when working from homewith a poor connection and when accessing folders with large numbers of files.

In these instances, turning off Optimize Connection Buffer Size in WinSCP can help:

• In WinSCP, goto the settings for the site (ie. from the menu Session->Sites->SiteManager)

• From the Site Manager dialog click on the selected session and click edit button

• Click the advanced button

• The Advanced Site Settings dialog opens.

• Click on connection

• Untick the box which says Optimize Connection Buffer Size

Strange fonts or errors re missing fonts when trying to start a graphical application

Certain programs require esoteric fonts to be installed on the machine running the X server (i.e. your local machine).Example of such programs are qmon, a graphical interface to the Grid Engine scheduling software, and Ansys. If youtry to run qmon or Ansys on a Linux machine and see strange symbols instead of the latin alphabet or get an errormessage that includes:

X Error of failed request: BadName (named color or font does not exist)

then you should try running the following on your own machine:

for i in 75dpi 100dpi; dosudo apt-get install xfonts-75dpipushd /usr/share/fonts/X11/$i/sudo mkfontdirpopdxset fp+ /usr/share/fonts/X11/$i

done

Note that these instructions are Ubuntu/Debian-specific; on other systems package names and paths may differ.

Next, try connecting to a cluster using ssh -X clustername, start a graphical session then try runningqmon/Ansys again. If you can now run qmon/Ansys without problems then you need to add two lines to the .xinitrc file in your home directory on your own machine so this solution will continue to work following a rebootof your machine:

FontPath /usr/share/fonts/X11/100dpiFontPath /usr/share/fonts/X11/75dpi

Can I run programs that need to be able to talk to an audio device?

On ShARC all worker nodes have a dummy sound device installed (which is provided by a kernel module calledsnd_dummy).

This may be useful if you wish to run a program that expects to be able to output audio (and crashes if no sound deviceis found) but you don’t actually want to monitor that audio output.

snd_dummy is not (yet) set up on Iceberg’s worker nodes.


https://www.alsa-project.org/main/index.php/Matrix:Module-dummy


Login Nodes RSA Fingerprint

The RSA key fingerprint for Iceberg’s login nodes is:

de:72:72:e5:5b:fa:0f:96:03:d8:72:9f:02:d6:1d:fd

Issue when running multiple MPI jobs in sequence

If you have multiple mpirun commands in a single batch job submission script, you may find that one or more ofthese may fail after complaining about not being able to communicate with the orted daemon on other nodes. Thisappears to be something to do with multiple mpirun commands being called quickly in succession, and connectionsnot being pulled down and new connections established quickly enough.

Putting a sleep of e.g. 5s between mpirun commands seems to help here. i.e.

mpirun program1sleep 5smpirun program2

Warning about ‘groups: cannot find name for group ID xxxxx’

You may occasionally see warnings like the above e.g. when running a Singularity container or when running thestandard groups Linux utility. These warnings can be ignored.

The scheduler, Son of Grid Engine, dynamically creates a Unix group per job to keep track of resources (files andprocess) associated with that job. These groups have numeric IDs but no names, which can result in harmless warningmessages in certain circumstances.

See man 8 pam_sge-qrsh-setup for the details of how and why Grid Engine creates these groups.

Glossary of Terms

The worlds of scientific and high performance computing are full of technical jargon that can seem daunting to new-comers. This glossary attempts to explain some of the most common terms as quickly as possible.

HPC ‘High Performance Computing’. The exact definition of this term is sometimes a subject of great debate butfor the purposes of our services you can think of it as ‘Anything that requires more resources than a typicalhigh-spec desktop or laptop PC’.

GPU Acronymn for Graphics Processing Unit.

SGE Acronymn for Sun Grid Engine.

Sun Grid Engine The Sun Grid Engine is the software that controls and allocates resources on the system. Thereare many variants of ‘Sun Grid Engine’ in use worldwide and the variant in use at Sheffield is the Son of GridEngine

Wallclock time The actual time taken to complete a job as measured by a clock on the wall or your wristwatch.

2.6. Glossary of Terms 369





Index

GGPU, 369

HHPC, 369

SSGE, 369Sun Grid Engine, 369

WWallclock time, 369

371

Documents

Sheffield HPC Documentation Release