Spring Semester 2020 to MSI Spring 2020.pdf• OpenStack for Secure Cloud Web Portals & Databases • Galaxy for Multi-omics • Jupyter Hub. Minnesota Supercomputing Institute

Minnesota Supercomputing Institute

© 2019 Regents of the University of Minnesota. All rights reserved.

Introduction to MSI Spring Semester 2020

Shane Scottmsi.umn.edu → About → Staff



• Why: Mission • Who: Organization • What: Resources and Services • How: Eligibility, Access, and Accounts

Overview



• To provide advanced research computing infrastructure and expertise to the University of Minnesota research and scholarly community and the State of Minnesota in order to advance and accelerate research and foster innovation and discoveries through advanced computing technologies, scientific computing and informatics, application development, and services.

• Academic unit under the Office of the Vice President of Research• 42 full time employees and ~6 students

• Six working teams• 800+ active Principle Investigator groups • 4,600+ active users




Office of the Vice President for Research

Advanced Systems

Operations

- Common Services- HPC Systems

- Storage Systems- Hosted Services

Scientific Computing Solutions

-- Optimization -- Benchmarking-- HPC Research

Workflow & pipeline Development

Application Development

Solutions

- Custom App Dev

- System Programming

Research Informatics Solutions

-Informatics education

-Informatics research

-Informatics services

-Life Science Computing

User Gateway Group

-- User Support Lead

-- User Training-- On Boarding

-Communications-- Outreach

Minnesota Supercomputing

Institute

UMInformatics

InstituteU-Spatial

Research Computing

Group Distribution: 836



Service Units (SUs) by Discipline



Storage (TBs) by Discipline



Resources and Services



Batch HPC & Interactive HPC



Structure of MSI SystemsYou!



• Highly parallel computing environment with high-speed network connections, graphics processing units, high memory, and high-performance storage

• Offer both batch HPC and interactive HPC• HPC use requires service units (SUs)

High Performance Computing Services



• 741 nodes• 17,784 cores• 40 nodes with 2x

NVidia Tesla GPUs• 32 nodes with

480GB SSDs

HPC Systems

• 164 AMD ROME Nodes

• 20,992 cores• Amd2tb Nodes• v100 GPU nodes• v100-4 GPU nodes• v100-8 GPU nodes

Mesabi Mangi



Structure of HPCSystems



• Submit jobs to the cluster scheduler, close up your laptop or shut down your workstation, go home, analysis runs on the computer

• Requires jobs to be written as shell scripts and submitted via the command line

• Jobs get queued and run based on a combination of the size of the request, the current load of the systems, how many jobs you have run

• This is called “FairShare” - details can be found at this link:https://www.msi.umn.edu/content/hpc

• This will probably be where you spend most of your time interacting with MSI, unless you have a specialized interactive application

Batch HPC

https://www.msi.umn.edu/content/hpc



• To use batch HPC, you just submit your job to a queue• The size of the job (number of cores, amount of memory, length of

time) will determine the queue that you can use• For Mangi:

Scheduling and Queues



Job Submission and Scheduling (PBS Scripts)

#!/bin/bash -l#PBS -l walltime=1:00:00,nodes=1:ppn=8,pmem=220mb#PBS -m abe#PBS -M [email protected]

cd /home/msistaff/nscott/PBSExample

module load matlab

matlab -nodisplay -r "maxNumCompThreads($PBS_PPN); " < shanetest.mat

Batch HPC: PBS

https://www.msi.umn.edu/content/job-submission-and-scheduling-pbs-scripts



• MSI also offers interactive HPC access• Very useful for real-time data visualization and exploration• Allows use of specialized hardware and software without labs

having to purchase it• Types of interactive HPC offered:

• qsub -I: interactive shell on batch HPC queue• NICE: web-based desktop• CITRIX: remote Windows desktop on HPC hardware• NX NoMachine: remote Linux graphical desktop• Jupyter Notebooks (beta): interactive Python/R on MSI

Interactive HPC



Interactive queue use with qsub

ssh -X mesabi

nscott@ln0004 [~] % qsub -X -l nodes=1:ppn=4,mem=8gb,walltime=12:00:00 -q interactive -Iqsub: waiting for job 19427077.mesabim1.msi.umn.edu to startqsub: job 19427077.mesabim1.msi.umn.edu ready

nscott@cn0001 [~] % module load XXXXXX

Interactive HPC

https://www.msi.umn.edu/content/interactive-queue-use-qsub



• Software is loaded as a module• Common software tools used in many fields are available on our

systems• Sequence analysis• Genetics• Proteomics• Image processing• Computational chemistry• Etc…

• 500+ software applications• MSI Software Search

• We can work with your research groups to install software that you need

• Contact the help desk!

Software Resources

https://www.msi.umn.edu/software



• Represent CPU-time (cores * time) on the HPC systems• 1 SU = 1.5 CPU-hours

• Default allocation for a group is 70,000 SUs, but you can request more• For requests between 70,000 and 280,000 SUs:

• Justification if group has not previously been granted 280,000• For requests > 280,000 SUs:

• Specify amount of SUs required• Justification

• SUs renew with group renewal at the end of year• If you run out, you can file a supplemental allocation request

Service Units



Data Storage Solutions



• Primary storage is where you will do most of your work• GNU/Linux filesystem

• Watch your disk usage:• If you are up against file number or disk usage quota, your entire

group cannot make new files• Important directories in your primary storage space:

/home/groupname/YOUR_X.500Your home directory. Store scripts and small files here

/home/groupname/sharedPI group shared directory. Store shared software, common datasets, reference databases, etc, here.

Primary Storage



• Quotas are very high, so there should be leeway to make very large files here

• Use it for intermediate/temporary files in analytical workflows• Files are kept for 30 days, so be sure to copy important files to your

group directory• Scratch space is also not backed up by snapshot• We know the tricks that exist to keep data in scratch for

longer than 30 days. • Location:

/panfs/roc/scratch/Or/scratch.global/

Primary StorageScratch



• Second tier storage is separate from primary storage - you must use special tools to access it

• From the command line, you can use the “s3cmd” software package to interact with it

• See: https://www.msi.umn.edu/support/faq/how-do-i-use-second-tier-storage-command-line

• For a graphical interface, you can use Globus• See: https://www.msi.umn.edu/support/faq/how-do-i-use-globus-transfer-data-msi-0

• Not backed up, so if you delete a file, it will be gone for good• Quotas are much more relaxed on second tier storage• Please use it!

Second Tier StorageCeph

https://www.msi.umn.edu/support/faq/how-do-i-use-second-tier-storage-command-line

https://www.msi.umn.edu/support/faq/how-do-i-use-globus-transfer-data-msi-0



• Archive storage is tape backup• Very long term write-once storage - not made for “active” data• Data will not be erased, either• High volume: >10s of TB of data• tar (tape archive) your data for storage on tape

• We will work with you to get your data situated if you want to purchase archive storage

Archive Storage (Tape)



• Default storage allocation: 150GB primary storage• Requests of up to 1TB:

• No justification or review required• For requests of up to 5TB:

• Justification• Review by MSI staff

• For requests of more than 5TB:• Justification• Estimate of amount needed• Estimate of amount actively used• Estimate of duration of storage• Review by MSI staff

Data Storage Allocation

Recap: MSI Computing and Data Storage AssetsBatch High Performance Computing

• Two Supercomputers• 28,000 CPU Cores• Infiniband Network Big Data Storage & Analysis

• 6 PB Primary High Performance • 3 PB Second Tier • 30 PB Archive Tape Library

Interactive & Cloud Computing• Citrix VDI for Windows • DCS Nice for Linux Desktops• OpenStack for Secure Cloud

Web Portals & Databases• Galaxy for Multi-omics • Jupyter Hub



PI Groups, Accounts and Support



Service Costs• Free for UMN faculty and their associates• Free for other MN academic institutions• Fee for resources over allocations limits • Fee for hosting UMN faculty systems• Fee for software development• Fee for service for external organizations



PI Group Structure

• PI: Controls everything

• Group Administrators: Users within the group designated to do everything a PI can do except add other Group Administrators

• Users: Added by PI, usually lab members (postdocs, techs, undergrads, grad students)

• Users may be part of multiple accounts if they perform analysis for multiple PIs

• The PI is a user, too, and their SU and disk usage counts the same as their users’



Checking Group Usage

• Check SU usage:https://www.msi.umn.edu/group/GROUPNAME/su

• Check disk usage:https://www.msi.umn.edu/group/GROUPNAME/storage

https://www.msi.umn.edu/group/dhp

https://www.msi.umn.edu/group/GROUPNAME/storage



Accounts and Renewals

• UMN faculty members, department staff, and researchers at other MN state institutions are eligible for MSI accounts

• PI accounts are renewed annually• Renewal period starts in October

• Small supplemental allocation requests can be handled through help desk

• “Small” is below the bolded thresholds shown in the previous slides

• Committee to approve large requests meets several times per year



• MSI has 24 consulting staff• 17 consultants have an advanced degree• We provide 10-20 workshops and tutorials annually• 1-2 hour meetings on computational or informatics methodologies• Informational sessions for proposal development• Expertise including:

• Research Computing; Informatics• Big Data Analysis• Parallel Algorithm Optimization and Development• Custom Analysis Pipeline Development• Genomics• Custom Application Development

Research SupportServices



• We host tutorials on a range of topicshttps://www.msi.umn.edu/tutorials

• Introductory tutorials• Intro to MSI• Intro to Linux• Intro to Job Submission

• Bioinformatics tutorials• Proteomic and mass spec data analysis• RNAseq data analysis

• Advanced tutorials• Parallelization• Python for scientific computing

• And many more!

Training and Tutorials

https://www.msi.umn.edu/tutorials



Links to Resources• MSI Homepage

• https://www.msi.umn.edu• PI eligibility and access policies

• https://www.msi.umn.edu/content/eligibility-getting-access• Queue limits page

• https://www.msi.umn.edu/queues• Tutorials page

• https://www.msi.umn.edu/tutorials• Check SU balance

• https://www.msi.umn.edu/group/GROUPNAME/su• Check storage

• https://www.msi.umn.edu/group/GROUPNAME/storage• Software tools

• https://www.msi.umn.edu/software

https://www.msi.umn.edu/

https://www.msi.umn.edu/content/eligibility-getting-access

https://www.msi.umn.edu/queues

https://www.msi.umn.edu/tutorials

https://www.msi.umn.edu/software



Questions?• MSI Help Desk

– Monday-Friday– 9:00 AM – 4:00 PM

• Walk-in help is available in Walter 587• Call x6-0802 (1-612-626-0802)• Email [email protected]• Schedule Zoom Consultation

https://www.msi.umn.edu/content/helpdesk-video-consultations

Documents

Spring Semester 2020 to MSI Spring 2020.pdf• OpenStack for Secure Cloud Web Portals & Databases • Galaxy for Multi-omics • Jupyter Hub. Minnesota Supercomputing Institute