Overview of Software Environment on Fugaku

Overview of Software Environment

on Fugaku Hitoshi MuraiRIKEN R-CCS

June 30, 2021 16th meeting for application code tuning on

A64FX computer systems.

Introduction• Programming Env. WG in FS2020 project

• responsible for designing the programming env. of Fugaku.• languages & compilers• tools (performance profiler, debugger, etc.)

• based on the knowledge learned from our experience of the K computer and requests from the pilot users.

• Goals:• higher performance, efficiency, and usability of Fugaku.• contribution to the software ecosytem of Arm processors.

June 30, 2021 26th meeting for application code tuning on

A64FX computer systems.

Basic Concept of the Design• The vendor (Fujitsu) provides essential software.

• compilers• libraries (e.g. MPI, BLAS)• tools

• We provide Open Source Software (OSS).• as supplements or alternatives to vendor-provided ones.• through Spack

June 30, 20216th meeting for application code tuning on

A64FX computer systems. 3

Languages & Compilers (for Fugaku)



Version Languages OpenMP TypeFujitsu 4.5.0 F2008, C11, C++14 4.0+ native & crossGCC 8 & 10† F2008-, C17, C++17+ 4.5+ native & WIP for crossLLVM 10‡ & 11+† F2008-, C11, C++17+ 4.0+ native

† provided through Spack‡ available on the Arm compile node

MPI



Spec. Thread Support Level Optimized for Fugaku Compiler Note

Fujitsu MPI 3.1+ MPI_THREAD_SERIALIZED Yes Fujitsu or GCC† based on Open MPI

MPICH on Tofu 3.0 MPI_THREAD_MULTIPLE Not fully Fujitsu (WIP) or GCCprovided as a Spackpackage "mpich-tofu"

† See Section 4 of Supercomputer Fugaku Users Guide - Language and development environment - Version 1.05.

Math Library• Fujitsu provides:

• SSL II / C-SSL II (Fujitsu Scientific Subroutine Library II),• BLAS / LAPACK / ScaLAPACK,• High-speed quadruple precision basic arithmetic library.• They are fully optimized for Fugaku.

• We provide many others through Spack, including:• FFTW (fujitsu-fftw), optimized for Fugaku,• PETSC,• EigenExa,• etc.



Vendor-provided Tools• Integrated Development Environment (based on Eclipse)• Performance profilers

• Instant Performance Profiler (FIPP) ― profiles overall performance based on sampling.

• Advanced Performance Profiler (FAPP) ― profiles regional performance based on manual instrumentation of programs.

• CPU Performance Analysis Report ― visualizes FAPP outputs as tables and graphs in MS Excel.

• Debugging features• abnormal termination inspection• deadlock inspection• scripted debugging



Other Tools• OSS (ported and maintained in collaborative works)

• Score-P/Scalasca @ JSC ― performance instrumentation, measurement, trace analyzer (WIP)

• TAU @ U. Oregon ― profiling and tracing toolkit for performance analysis of parallel programs (WIP)

• VeloC @ ANL ― very low overhead transparent multilevel checkpoint/restart

• Commercial Software• Vampir• Arm Forge (WIP)



Visualization Tools• The following usage scenarios (tools) for visualization are

available on Fugaku.• CLI (Command Line Interface)

ImageMagick, POV-Ray, GrADS• GUI (Graphical User Interface)

ImageMagick, Ncview, ParaView• Distributed (Client / Server)

ParaView• VNC (GNOME Desktop)

• Refer to Fugaku Visualization User Guide for more detail.



Basic Policy about OSS on Fugaku• We select and provide ones from the most widely-used

OSS packages;• Users are able to install packages for themselves if needed.• The above two are achieved with Spack.• Best effort support:

• No specific supports for packages are provided in principle.• Packages are provided as-is (not optimized).• not fully tested.



Spack• A package management tool designed for large

supercomputing centers.• Developed by LLNL in Exascale Computing Project (ECP).• Features:

• supporting multiple versions and configurations of software on a wide variety of platforms and environments including Fugaku (A64FX),

• >5000 recipes of packages available,• handling dependencies between packages.



spack-mirror.r-ccs.riken.jp/oss/public/

For the compatibility of OSS with Fugaku (A64FX), visit:

Spack on Fugaku (1)• Basic usage

• System Spack (public instance) + User Spack (private instance)

• The operators provides important or widely-used packages in the system Spack.

• Users can add their necessary packages in their own Spack.• These two instances can be "chained."



$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh$ spack load xxx

// setup Spack// load a package XXX

Spack on Fugaku (2)• Some very important packages (e.g. FFTW) have been

optimized for Fugaku.• Fore more detail, refer to the documents available on the

Fugaku portal.• Resource – Use of OSS• Resource – Manuals – User's Guide – Fugaku Spack User Guide

• Important Notice:• Spack does not support cross compilation yet!• You have to install packages for Fugaku on the compute nodes.



Available Packages on Compute Nodes

June 30, 2021 14

Name Versionadios2 2.6.0

alps 2.3.0bcftools 1.10.2

bedtools2 2.27.1blitz 1.0.1boost 1.74.0

bwa 0.7.17cblas 2015-06-06cmake 3.18.4dssp 3.1.4

eigenexa 2.6fujitsu-fftw masterfujitsu-mpi 4.3.1

fujitsu-ssl2 4.3.1gcc 10.2.0

genesis 1.5.1grads 2.2.1

gromacs 2021.1hdf5 1.10.7

htslib 1.10.2julia 1.5.2

kokkos 3.1.01lammps 20201029

Name Versionlis 2.0.27

llvm 11.0.0lz4 1.9.2

mapslice2 2.2.1metis 5.1.0

mpich-tofu mastermptensor 0.3.0

n2p2 2.1.1netcdf-c 4.7.4

netcdf-cxx 4.2netcdf-cxx4 4.3.1

netcdf-fortran 4.5.3netlib-

scalapack 2.1.0

openblas 0.3.12openfoam 2006

openfoam-org 8openjdk 11.0.0-2020-01-01

parallel-netcdf 1.12.1paraview 5.8.1parmetis 4.0.3

petsc 3.14.1pfapack 41899picard 2.20.8povray 3.7.0.8

Name Versionprocess-in-

process 2 & 3

py-dask 2.16.0py-h5py 2.10.0

py-jupyterhub 1.0.0py-keras 2.2.4

py-mpi4py 3.0.3py-netcdf4 1.5.3

py-numpy 1.19.4py-pydmd 0.3py-pygps 1.3.5py-pysam 0.15.2

py-scikit-learn 0.23.2py-scipy 1.5.4

py-seaborn 0.9.0py-xarray 0.14.0

quantum-espresso 6.6r 4.0.3

raja 0.12.1samtools 1.1

screen 4.8.0star 2.7.6atmux 3.1bxios 2.5

Note: many other ones on which these packages are depend are also available.

R-CCS Software & Supported Software• R-CCS Software

• Process-in-Process (PiP)• MPICH on Tofu• Omni XcalableMP• EigenExa• GENESIS• NTChem• SCALE (TBA)• McKernel (TBA)



• Supported Software• ABINIT-MP

Fore more detail of each software, visit the page on the Fugaku portal.

AI Frameworks• PyTorch 1.7.0

• to be updated to the latest around September.• TensorFlow 2.2.0

• to be updated to the latest around September.• Chainer 4.5.0

• These are fully optimized for Fugaku in the DL4Fugaku project and currently provided w/o Spack.• Please contact to the Fugaku support desk if interested.



• Vendor-provided and open-source software complement each other to achieve higher efficiency and usability of Fugaku.• OSS packages are available through Spack, which allows

both the operators and users to easily manage them.• For some specific applications (e.g. visualization & deep

learning), we provide and/or optimize a set of software.



Summary

Documents

Overview of Software Environment on Fugaku