Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3...

OPENCCQFederated High Performance Computing with

Where to start with HPC in the Cloud?

Here – Combine all the services yourself?

Clus

ter i

s Rea

dy

In Minutes your Cluster is ready

Collaboration

• Have the ability to create collaborations

• Invite other collaborators to CloudyCluster

• Initially can share Google Drive Folders

• Oauth and Shib / InCommonsupport for Web Authentication

Self-Service Elastic HPC

Scheduler

CCQ

EFS S3

Auto-ScalingCompute

OrangeFSHPCParallelStorage

DDB

HPCJob

Login

WebDAViRODSGlobus

CreateafullyoperationalHPCClusterinminutes,completewith:• Storage: OrangeFSonEBS,S3,EFS

• Compute: JobDrivenElasticComputethroughCCQ

• Scheduler: withv.1.2Torque/Maui&SLURMwithCCQMetaScheduler

• HPCLibraries:Boost,Cuda Toolkit,Docker,FFTW,FLTK,GCC,Gengetopt,GRIB2,GSL,Hadoop,HDF5,ImageMagick,JasPer,NetCDF,NumPy,Octave,OpenCV,OpenMPI,PROJ,R,Rmpi,SciPy,SWIG,WGRIB,UDUNITS

• HPCSoftware:Ambertools,ANN,ATLAS,BLAS,Blast,Blender,Burrows-WheelerAligner,CESM,GROMACS,LAMMPS,NCAR,NCL,NCO,nwchem,OpenFoam,papi,paraview,QuantumEspresso,SAMtools,WRF

• YoucanalsoInstallyourowninacustomAMI• AllfromaneasytouseWebUIfrommobile,tabletordesktop• iRODS andXDMoD aretargetedforFallrelease.

Torque/MauiSlurm

CCQ - Elastic HPC Dispatching

Scheduler

DynamoDB

Login Instance

Public Subnet

Submit the Job

Through CCQ

CCQ holds job determines and

launches instances needed

CCQ Sends the job to the

scheduler when ready

Scheduler launches the Job normally

If no jobs are in the queue for that

instance type near the billing hour, instances

are terminated

Compute Groups

SchedulerTorqueSlurm

OpenCCQ Project (Federated HPC)On Prem to Cloud bursting using heuristics and hints to determine where the job goes; supporting multiple schedulers and environments

Job Routing• On Premise• Cloud• Nat. ResourcesData MovementJob to Data

For $100 AWS Credit you can fill out the form at:http://www.cloudycluster.com/ccvisualhpc/

Discussion and Questions?

Come see a demo at our table