Amsterdam, 28 June 2006DEISA UNICORE tutorial UNICORE and the DEISA supercomputing grid Jules Wolfrat [email protected]

Amsterdam, 28 June 2006 DEISA UNICORE tutorial

UNICORE and the DEISA supercomputing grid

Jules Wolfrat

[email protected]

Amsterdam, 28 June 2006 DEISA UNICORE tutorial 2

Outline

• DEISA overview• UNICORE history• UNICORE architecture• Demo?

Amsterdam, 28 June 2006 DEISA UNICORE tutorial 3GEAN

T

AIX distributedsuper-cluster

Vector systems(NEC, …)

Linux systems(SGI, IBM, …)

THE DEISA SUPERCOMPUTING GRID


DEISA objectives

• To enable Europe’s terascale science by the integration of Europe’s most powerful supercomputing systems.

• Enabling scientific discovery across a broad spectrum of science and technology is the only criterion for success

• DEISA is an European Supercomputing Service built on top of existing national services. This service is based on the deployment and operation of a persistent, production quality, distributed supercomputing environment with continental scope.

• The integration of national facilities and services, together with innovative operational models, is expected to add substantial value to existing infrastructures.

• Main focus is High Performance Computing (HPC).


BSC Barcelona Supercomputing Centre Spain

CINECA Consortio Interuniversitario per il Calcolo Automatico Italy

CSC Finnish Information Technology Centre for Science Finland

EPCC/HPCx University of Edinburgh and CCLRC UK

ECMWF European Centre for Medium-Range Weather Forecast UK (int)

FZJ Research Centre Juelich Germany

HLRS High Performance Computing Centre Stuttgart Germany

IDRIS Institut du Développement et des Ressources France

en Informatique Scientifique - CNRS

LRZ Leibniz Rechenzentrum Munich Germany

RZG Rechenzentrum Garching of the Max Planck Society Germany

SARA Dutch National High Performance Computing The Netherlands

and Networking centre

Participating Sites


The DEISA supercomputing environment(21.900 processors and 145 Tf in 2006, more than 190 Tf in 2007)

• IBM AIX Super-cluster – FZJ-Julich, 1312 processors, 8,9 teraflops peak

– RZG – Garching, 748 processors, 3,8 teraflops peak

– IDRIS, 1024 processors, 6.7 teraflops peak

– CINECA, 512 processors, 2,6 teraflops peak

– CSC, 512 processors, 2,6 teraflops peak

– ECMWF, 2 systems of 2276 processors each, 33 teraflops peak

– HPCx, 1536 processors, 11 teraflops peak

• BSC, IBM PowerPC Linux system (MareNostrum) 4864 processeurs, 40 teraflops peak

• SARA, SGI ALTIX Linux system, 416 processors, 2,2 teraflops peak

• LRZ, Linux cluster (2.7 teraflops) moving to SGI ALTIX system (5120 processors and 33 teraflops peak in 2006, 70 teraflops peak in 2007)

• HLRS, NEC SX8 vector system, 576 processors, 12,7 teraflops peak.

• Systems interconnected with dedicated 1Gb/s network – currently upgrading to 10 Gb/s – provided by GEANT and NRENs


The technology cycle

Technology providersR&D projects

DEISA strategic andtechnologic management

Service definitionsTechnology specifications

Technology watch

Technology pull

WAN GPFS (IBM) completedMulti-cluster batch processing (IBM) completedGPFS for non-IBM systems (IBM) ongoingCo-scheduling (Platform) in preparation


How is DEISA enhancing HPC services in Europe?

• Running larger parallel applications in individual sites, by a cooperative reorganization of the global computational workload on the whole infrastructure, or by the operation of the job migration service inside the AIX super-cluster.

• Enabling workflow applications with UNICORE (complex applicaions that are pipelined over several computing platforms)

• Enabling coupled multiphysics Grid applications (when it makes sense)

• Providing a global data management service whose primordial objectives are:

– Integrating distributed data with distributed computing platforms– Enabling efficient, high performance access to remote datasets (with Global File

Systems and striped GridFTP). We believe that this service is critical for the operation of (possible) future European petascale systems

– Integrating hierarchical storage management and databases in the supercomputing Grid.

• Deploying portals as a way to hide complex environments to new users communities, and to interoperate with another existing grid infrastructures.


Basic Services: Global File Systems

Global file system

Sophisticated software environment,necessary to provide single systemimage if a clustered computingplatform.

They provide global data management.Data in the GFS is “symmetric” withrespect to all computing nodes.

HPC system at site A

nodes

Disk space

network


The DEISA integration concept

Global distributed GPFS file system with continental scope.Global resource pool is dynamic: nodescan enter and leave the pool withoutDisrupting the national services.

Site A

Site C Site D

Network interconnect(Reserved bandwidth)

Site B


Linux SGI

SARA (NL)

LRZ (DE)LRZ (DE)

DEISA Global File System integration in 2006(based on IBM’s GPFS)

CINECA (IT)FZJ (DE)

ECMWF (UK) IDRIS (FR)

AIX IBM domain

RZG (DE)

BSC (ES)

LINUX Power-PC

CSC (FI)

HPC Common Global File System similar architectures / operation systems

High bandwidth (10 Gbit/s)

High Performance Common Global File System various architectures / operating systems

High bandwidth (up to 10 Gbit/s)


Demonstration of a transparent data access in a heterogeneous

configuration

)

(1) A 64 processor job is running at SARA (SGI Altix system)

Global File System

(2) The input data for this run are read from the Linux GPFS at SARA

(3) The output data will be written into the BSC GPFS system in Spain

(4) Visualization at RZG system is reading the output data produced by the application from BSC GPFS

Demonstration

RZG (G)

BSC (ES)

SARA (NL)


Amsterdam, NL

Orsay,FR Garching,DE

Bologna, IT

Jülich, DE

Argonne, IL

Bloomington , INUrbana-

Champaign , ILSan Diego, CA

TeraGrid Sites

DEISA Sites

American and European supercomputing infrastructures linked: bridging communities with scalable, wide-area global file systems

Global File System Interoperability demo duringSupercomputing Conference 2005 in Seattle


Basic services: workflow simulations using UNICORE

UNICORE supports complex simulations that are pipelined over several heterogeneous platforms (workflows).

UNICORE handles workflows asa unique job and transparently moves the output – input data along the pipeline.

UNICORE clients that monitorthe application can run in laptops.

UNICORE has a user friendly graphical interface. DEISA has developed a command line interface for UNICORE.

UNICORE infrastructure including all sites has full production status.It has proven to be very stable during the last few months.


Other basic services

• Job migration inside the AIX super-cluster. Based on LoadLeveler Multi-Cluster, it allows system administrators to reroute jobs to other sites, in a way transparent for the end users. Used to move away simple jobs of « implicit users » to make place for a bigger application in a site. Full production status.

• Co-allocation. We are starting to prepare a first generation co-allocation service on the full heterogeneous infrastructure, using LSF Multi-cluster. Important for coupled Grid aplications and for data movement. Service in development phase, prototype expected in 6-9 months

• Remote I/O using Global File Systems and fast data transfers. See next transparency

• Integrating hierarchical data management and databases in the supercomputing Grid. In progress


Accessing remote data: high performance remote I/O and file transfer

GridFTP

Co-scheduled, parallel datamover tasks

DATA REPOSITORY

Remote I/O with globalfile systems implicitlymoves data across platforms(in production today)

DEISA will also deployexplicit high performancedata movers, using GridFTP


Summary

• DEISA provides an integrated supercomputing environment, with efficient data sharing through high performance global file systems. This is highly transparent to end users.

• DEISA enables job migration across sites (also transparent to end users). Exceptional resources for very demanding applications are made available by the operation of the global resource pool. We are load balancing computational workload at a European scale.

• Huge, demanding applications can be run “as such”.

• Support of Grid applications (which are distributed by design).

• With this operational model, the DEISA super-cluster is not very different from a “true” monolithic European supercomputer (which must be partitioned in any case for fault tolerance and QoS).

• The main difference comes from the coexistence of several independent administration domains. This requires, as in TeraGrid, coordinated production environments.


UNICORE

UNICORE

UNiform Interface to COmputer Resources

Following material thanks to UNICORE team


Highlights• Excellent workflow support• Transparent data staging / transfer• Multi-site, multi-step jobs: heterogeneous meta-computing• Uniform user authentication and security mechanisms• The site maintains full control over their resources• UNICORE Client offers

– Uniform GUI for job creation and monitoring

– Easy integration of applications through plugins


History I• Development started in 1997• Projects UNICORE and UNICORE Plus

– Funded by the German Ministry of Education and Research (until 12/2002)


History IIDevelopments in EC funded projects: • EUROGRID (11/2000 – 01/2004)

– IST-1999-20247– Resource broker, Standard based File Transfer (gridFTP)– Bio molecular simulations, Weather prediction, coupled CAE simulations, Structural analysis

• GRIP (01/2002 – 02/2004)– IST-2000-32257– Interoperability between UNICORE and Globus (Integration of Globus maintained resources as

target system in UNICORE)

• OpenMolGRID (09/2002 – 11/2004)– IST-2001-37238– Use UNICORE for molecular engineering– Focus on scientific workflows


History III

• Collaborators:– Intel GmbH (former Pallas GmbH)– Fujitsu Laboratory of Europe (former fecit)– Forschungszentrum Jülich– Deutscher Wetterdienst– Genias, RUS, RUKA, LRZ, PC2, ZHR, ZIB– CNRS-IDRIS (F), CSCS (CH),

GIE EADS CCR (F), ICM (PL), Parallab (N), Soton (UK), UoM (UK), ANL (US), UT (EE), UU (UK), ComGenex (HU), Negri (I)


• Features– Intuitive GUI with single sign-on– X.509 certificates for AA and job/data signing– only one opened port in firewall required– workflow engine for

• complex multi-site • multi-step workflows• job monitoring

– extensible application support – secure data transfer integrated– resource management– easy installation and configuration of client and server components– full control of resources remains– production quality, …


Software Status I

• Current version 5.6 (Client) / 4.6 (Server)• User Client is platform independent (Java)• Servers (Unix)• Target systems (Unix)

– “no batch“

– T3E, SP3, VPP, hpcLine, SR 8000, SX-5, PC-Clusters, … , Globus 2.x as targets

– NQS, LL, LSF, PBS, CCS, SGE, ...


Software Status II

• UNICORE available at SourceForge as OpenSource under BSD license

• http://unicore.sourceforge.net

• UNICORE Forum e.V.

• http://www.unicore.org

• Public test system for testing (standard) client functions available


Deployment

• At all project partner sites

• DEISA sites (IDRIS, CINECA, RZ Garching, ...)

• Naregi project (Japan)

• …


The UNICORE Grid

UNICORE Client

UNICORE Architecture

Amsterdam, 28 June 2006 DEISA UNICORE tutorial 28UsiteUsite

VsiteVsite

ARCHITECTURE

TSI

NJS

RMS

TSI

NJSAuthorization

GatewayAuthentication

opt. Firewall

Gateway

opt. Firewall

Client

Multi-Site Jobs

UUDB

SSL

Abstract

Non-Abstract

Disc RMS Disc

Vsite

TSI

NJS

RMS

UUDB

Disc

IDBIDB IDBIncarnation

opt. Firewall

Authorization


Client

JobPreparation

JobMonitoring

WorkflowManagement

Usites

Vsites


UNICORE Server• Gateway• Network Job Supervisor

– Configuration

– UNICORE User Data Base

• Target System Interface• Demo package containing preconfigured components available

onsourceforge.net/projects/unicore


Server Components

Network Job Supervisor

Gateway

UNICORE User DB

Target System Interface

conf

conf

UUDB


Server Prerequisites

• Gateway and NJS:– Java 1.4.2

– X.509 certificates for Gateway and NJS

– Signer certificate(s)

• TSI:– Perl ( 5.004)


Gateway

• Entry point of a UNICORE Site• Accepts SSL connections from Clients and NJSs• Accepts valid certificates from all signers known to it

(authentication)• Talks UNICORE Protocol Layer (UPL) on connections to the

outside world• Sends/receives AJOs to/from the NJSs


Gateway connections

Network Job Supervisor

Gateway

UNICORE User DB

conf

conf

UUDB

connections<Vsite name> <NJS machine> <NJS port>

gateway.properties gw.gateway_host_name=<host name> gw.port=<port>


Network Job Supervisor (NJS)

• UNICORE scheduler• Receives/sends AJOs from/to local Gateway • Translates AJO into batch job for target • Maps the user’s Ulogin to Xlogin• Sends sub-AJOs to corresponding Gateway according to

dependencies• Polls for status and output of sub-AJOs• Sends batch jobs and requests to TSI• Polls TSI for job status and output


NJS Connections

NJS

Gateway

UNICORE User DB

conf

conf

UUDB

njs.properties njs.gateway=<host name> njs.vsite_name=<name> njs.gateway_port=<port> njs.admin_port=<port>

connections

TSI

Admin


Incarnation Data Base

Static definitions and translation table, contains definitions for• GENERAL properties (file spaces, descriptions, …)

• EXECUTION_TSI (host + ports, resources, batch queues, …)

• STORAGE_TSI (for file transfers and management)

• RUN (translation rules for target)

• IMPORT, EXPORT, CLEANUP, LIST_DIRECTORY, RENAME, COPY_FILE, DELETE_FILE, CHANGE_PERMISSIONS

• FORTRAN, LINK


UNICORE User Data Base

• Management of Ulogin – Xlogin mapping information

• NJS accesses this information

• Basic version allows to map one certificate to exactly one Xlogin

• NJS to UUDB interface defined to adapt to site specific user data bases (i.e. ldap)

• http://www.unicore.org/downloads.htm contributions offers an alternative uudb with certificate-projectid pairs being mapped to Xlogins


NJS connections

NJS

Gateway

UNICORE User DB

conf

conf

UUDB

njs.idbSOURCE <TSI machine> <port1> <port2>

njs.properties njs.gateway=<host name> njs.vsite_name=<name> njs.gateway_port=<port> njs.admin_port=<port>

connections

TSI

Admin


Target System Interface

• Interface to target operating and batch system

• Perl scripts and modules

• Needs root privileges to act on behalf of the user (uses setreuid)

• Provides interface to local system for– Job submission

– Status query, job monitoring

– File handling

– …


Example: Submit.pm

$jobname = $2 if $1 eq "JOBNAME";$outcome_dir = $2 if $1 eq "OUTCOME_DIR"; $uspace_dir = $2 if $1 eq "USPACE_DIR";$time = $2 if $1 eq "TIME";$memory = $2 if $1 eq "MEMORY";$nodes = $2 if $1 eq "NODES";…$memory = "-lM $memory"."Mb";my $command = "$main::submit_cmd $queue $nodes $email $memory

$time $jobname $stdout_loc $stderr_loc $Submit::tsi_unique_file_name";


TSI connections

NJS

UNICORE User DB

conf

UUDB

njs.idbSOURCE <TSI machine> <port1> <port2>

njs.properties

TSI tsi.properties$main::njs_machine = shift || "NJS host";$main::njs_port = shift || "port1";$main::my_port = shift || “ port2”;


Overview: Server connections

Gateway

Admin

Client

Client

Client

SSL

SSL orplain socket

NJS

NJS TSI

TSIPlain sockets

njs.gateway_port+ GW connections file

gw.port+ Client Usite list

idb: SOURCE host p1 p2 + tsi script

njs.admin_port


Firewall Issues• Client Gateway

– Internet– Allow connections to Gateway for https protocol on the port the Gateway is listening on– Client side has to allow for outgoing traffic on any port

• Gateway NJS– Intranet– All connections from Gateway to NJS system and NJS’s Gateway port

• NJS TSI– Intranet– All connections from NJS to TSI system and TSI’s NJS port– All connections from TSI to NJS system and NJS’s TSI port


Current Trends and a look into the UNICORE future...

• Web services for interoperability• “open up“ the architecture• ...but keep the UNICORE strenghts

– abstraction and virtualisation

– workflows

– easy application integration


Acronyms I

UNICORE JobUjob

File space on the computer where Client runsNspace

File space at the VsiteXspace

Temporary file space for Ujob at VsiteUspace

Unix Login at VsiteXlogin

UNICORE Login, X.509 certificateUlogin

Computing resource, target systemVsite

Site providing UNICORE servicesUsite


Transl. of AJO into batch job using IDBIncarnation

Abstract Job ObjectAJOUNICORE Protocol Layer (Client – Gateway)UPL

Job Monitor Controller, part of ClientJMC

Job Preparation Agent, part of ClientJPA

Target System InterfaceTSI

UNICORE User Data BaseUUDB

Incarnation Data BaseIDB

Network Job SupervisorNJS

Acronyms II


meets

FZJ users

DEISA FZJ gateway

DMZ

FZJ NJS

intranet

CNE users

DEISA CNE gateway

DMZ

CNE NJS

intranet

RZG users

DEISA RZG gateway

DMZ

RZG NJS

intranet

IDR users

DEISA IDR gateway

DMZ

IDR NJS

intranet

CSC users

DEISA CSC gateway

DMZ

CSC NJS

intranet

SARA users

DEISA SARA gateway

DMZ

SARA NJS

intranet

BSC users

DEISA BSC gateway

DMZ

BSC NJS

intranet

LRZ users

DEISA LRZ gateway

DMZ

LRZ NJS

intranet

RZG users


UNICORE Security

• Security model based on X509 public key infrastructure• Credential consists of a public and a private key• No userid and password authentication• Password protected keystore• Single sign on• UNICORE accepts following private key formats:

– RSA (pkcs12)• E.g. Openssl 0.9.7x

– Java keystore (jks)• SUN Java

• Certificates provided e.g. by DFN CA• Two server site security entities:

– Gateway – Authentication– NJS – Authorisation


UNICORE Security - Client

• Access to password protected keystore• Encrypted Keystore contains all imported certificate(s) and the

user‘s private key(s)• UNICORE Keystore editor allows to

– Generate a X509 certificate request– Import/export .p12 or .jks keystores– Import public keys

• The User has to import (at least) three certificates into the Client– Pluginsigner‘s certificate (public key)– Gateway signer‘s certificate (public key)– User‘s signed public key


UNICORE Security: Gateway

• Gateway authenticates the user• Following checks are performed on certificates presented by a

client– Certificate is issued by one of the trusted CA (e.g. DFN-CA)– Certificate is within its validity period– Certificate has not been revoked (if check for Certification Revocation Lists

(CRL) is activated)

• Gateway accepts only SSL connections from Clients and other NJSs

– SSL-Handshake

• Optional SSL connection between Gateway and NJS


Behind the scenes: Authentication

Gateway

establish SSL connection

send user certificate

send gateway certificate

Trust user certificate issuer?

Trust gateway certificate issuer?

Gateway Certificate

Client

User Certificate


UNICORE Security: NJS

• NJS authorizes the user• Access the UNICORE user Database (UUDB)

– Maps the user‘s certificate to his xlogin on the target system

• Only users presenting certificates stored in the UUDB can connect to the target system

• NJS authorises other NJSs• Explicit UUDB entry


Behind the Scenes: Authorisation

IDB

TSI

UUDB

Certificate 2

Certificate 3

Certificate 4

Certificate 5

Certificate 1

Login B

Login C

Login D

Login E

Login A

Typical UNICORE

User

User Certificate

User Login

Client

NJS

Gateway

User Certificate

AJO


UNICORE Job

• Job contains – Sub-jobs and tasks

– Dependency information• Without dependencies all tasks of a job are executed in “parallel”

– Workflow: doN, loops, if-then-else

– Target system location

• Tasks are translated into batch jobs for the destination system by the servers (NJSs)


Abstract Job Object (AJO)

• Abstract, target system independent representation of a job

• Specifies actions to be performed by UNICORE

– Execute task

– File transfer task

– Control task

• Contains dependency graph

• Contains resource requests (nodes, memory, time, ...)

• Contains data set descriptions for data to be streamed

• Realised as Java classes


@

• Open Source under BSD license• Supported by FZJ

– Integration of own results andfrom other projects

– Release Management– Problem tracking– CVS, Mailing Lists– Documentation– Assistance

• Viable basis for many projects– DEISA, UniGrids, NaReGI, …

• http://unicore.sourceforge.net

Documents

Amsterdam, 28 June 2006DEISA UNICORE tutorial UNICORE and the DEISA supercomputing grid Jules Wolfrat [email protected]