IBM India

© 2007 IBM Corporation

Compiling and Executing Applications on BlueGene

Subba R Bodda and Ravi K Komanduri

IBM India Systems and Technology Laboratory

2

IBM India

© 2007 IBM Corporation

Agenda

BlueGene setup

Compiling applications on BlueGene

Executing applications on BlueGene

3

IBM India

© 2007 IBM Corporation

Redbooks

Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene

Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00

4

IBM India

© 2007 IBM Corporation

BlueGene setup

bglfen01(10.16.28.30),bglfen02 is the front end node bglsn is the service node Login to bglfen and compile your code Use mpirun or LL to submit your jobs bglsn has the service daemons which would spawn your jobs on actual BG nodes

5

IBM India

© 2007 IBM Corporation

IBM Compilers for Blue Gene

IBM compilers (Fortran, C, C++) for Blue Gene are located on the frontend node in /opt/ibmcmp

Fortran:

/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf

/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf90

/opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf95 C:

/opt/ibmcmp/vac/bg/8.0/bin/blrts_xlc C++:

/opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlC

Note : Users can add the respective directories to their PATH.

6

IBM India

© 2007 IBM Corporation

GNU Compilers for Blue Gene

GNU compilers (Fortran, C, C++) for Blue Gene are located in /opt/blrts-gnu/

Fortran:

/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g77

C:

/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-gcc

C++:

/opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g++

7

IBM India

© 2007 IBM Corporation

MPI on Blue Gene

MPI implementation on Blue Gene is based on MPICH-2 from Argonne

National Laboratory.

Include files mpi.h and mpif.h are at the location:

-I/bgl/BlueLight/ppcfloor/bglsys/include

Libraries to link for MPI are:

-L/bgl/BlueLight/ppcfloorbglsys/lib -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts

Note : The linking order for the MPI libraries should be as mentioned above.

8

IBM India

© 2007 IBM Corporation

Compiler Optimizations

9

IBM India

© 2007 IBM Corporation

Compiler Optimizations

10

IBM India

© 2007 IBM Corporation

IBM ESSL libraries

The IBM ESSL (Engineering Scientific Subroutine Library) is located in

/opt/ibmmath/essl/4.2/

11

IBM India

© 2007 IBM Corporation

Executing jobs using mpirun

cd /gpfs/fcHome/scratch/userid

mpirun –np 32 –partition R000 -cwd /gpfs/fcHome/scratch/ -exe /gpfs/fcHome/scratch/hello_c

Common options:

-args : List of arguments to the executables in double quotes

-env : List of environment variables in double quotes. “VARIABLE=value”

-mode : CO or VN (coprocessor/virtual-node)

-only_test_protocol : This options is used only to test communication protocol coprocessor mode : one MPI process per node, 1 GB limit per process.

Default mode virtual-node mode : two MPI processes per node, 512 MB limit per process; L3

cache, memory, networks, are shared

For more details type the following at a command prompt : mpirun -h

Note : A user can have only one job per Blue Gene partition

12

IBM India

© 2007 IBM Corporation

Executing jobs using submit_job Login to bglsn cd /bgl/BlueLight/ppcfloor/bglsys/bin ./mmcs_db_console mmcs$ allocate M03B_J214_64 mmcs$ submitjob M03B_J214_64 /bgl/hello.rts /bgl/ OK jobId=25 mmcs$ waitjob M03B_J214_64 25 OK mmcs$ free M03B_J214_64 OK mmcs$ quit

13

IBM India

© 2007 IBM Corporation

Environment Variables Environment variables can be specified using –env “XYZ=123 ABC=456”

Some of the environment variables are BGLMPI_EAGER BGL_APP_L1_WRITE_THROUGH BGL_APP_L1_SWOA BGLMPI_COPRO_SENDS BGLMPI_INTERRUPT

14

IBM India

© 2007 IBM Corporation

Redbooks

Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene

Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00

15

IBM India

© 2007 IBM Corporation

Summary

Compilation IBM XLF/XLC and GNU Compilers ESSL Libraries submit_job mpirun LL Environment Variables