Current Status of High Performance Computing and

CMIP6 at CMA

WANG Bin, WU Huanping, Xin XiaogeNMIC/CMA, NCC/CMA

Sept.12, 2019. ICAS 2019， Stresa, Italy.

Outline

• High Performance Computing at CMA

• Current Status of CMIP6 at CMA

NMIC in CMA——One of CMA’s National Operational Centers

Responsible for :

• Meteorological Communication

• Meteorological Data

• Meteorological Supercomputing

• Meteorological Administrative Affairs

Missions：

• Meteorological Communication Center: to collect and disseminate globally exchanged data, domestic observations and forecast products.

• Meteorological Data Center: to conduct life cycle data management (QC, Processing, Storage, Services and Archiving); develop atmospheric research datasets.

• High Performance Computing Center: to provide HPC resources and application support for operations, researches & development of numerical prediction models.

CMA Offices Met. Service

GovernmentDisaster Mitigation

Social Sectors Economy

CitizensBetter livelihood

High-Quality Data, Value-added ServiceFor Disaster Mitigation, Social Economy & Citizen Livelihood

VISION：

CMA Meteorological Data Center

CommunicationCenter

HPCCenter

Global Met. Data Domestic Met. Obs

Gov. Departments Crowd-sourcing Data

NWP Models Climate Models

Atmos. Reanalysis

Common Interface

Specialized Models

History of CMA HPC

PI-Sugon

• Peak Capability : 8,198.5 TFLOPS• Storage Capacity : 23,088 TB• CPU Cores : 98,432• PUE : 1.23

Two Subsystems backup each other:• Computing Nodes : 1504 • CPU + GPU• Intel MIC

• 2 Intel Xeon Processors per Node• 100Gb/s InfiniBand EDR network• Parastor 300 Parallel File System

Software Stack

IBM HPC PI-Sugon

Improvement

Peak Performance ~1000 TFLOPS ~8000 TFLOPS

Storage Capacity ~4000 TB ~18000 TB

Inter -Connection QDR 40Gb/s EDR 100Gb/s

Difference

OS AIX 7.1.0.0 RedHat Enterprise 7.4

并行文件系统 IBM GPFS Sugon Parastor

作业调度软件 IBM LSF（Loadleveler） Sugon Gridview（Slurm）

集群管理 IBM xCAT Sugon Clusconf

应用软件巨大差异！！！

编译器IBM XL C/C++ Compiler V11;

IBM XL Fortran Compiler V13.1

GNU（C/C++/Fortran等）

Intel Parallel Studio XE Cluster Edition

PGI编译器

并行环境 IBM Parallel Environment Runtime EditionOpenMP

OpenMPI，MPICH2，Mvapich2等

IBM & PI-Sugon

PI-Sugon Resource Usage

GRAPES & BCC_CSM

• GRAPES = Global/Regional Assimilation PrEdictionSystem

• BCC_CSM = Beijing Climate Center Climate System Model

GRAPES-GFS GRAPES_MESO GRAPES-TYM GRAPES-GEPS GRAPES-MEPS

Forecast range 10d 1.5d 5d 15d 3.5d

Domain Global China West Pacific Global China

H-resolution 25KM 3KM 9KM 50KM 10KM

V-resolution60L

3hPa

50L

10hPa

68L

10hPa60L 3hpa

50L

10hPa

Forecast time00,12 UTC 240h

06,18 UTC 120h00,06,12,18 UTC 00,12 UTC

00,12UTC

31members

00,12 UTC

15 members

Operation Runtime Schedule

12

• High performance computer management software

• Refined resource management system

• Operational monitoring system

• Numerical Model R & D Supporting Software

• Code management system

• GRAPES Integrated Setting Experiment Tool(GISET)

Model-Supportive Software Systems

Refined resource management system

• Resource management of IBM & PI-Sugon systems

• Unified management of national and regional resources

• Real-time and historical statistical analysis of system resource usage and utilization

• Computing resource and storage resource usage accounting

• Model & job statistical analysis

• Monitoring of IBM & PI-Sugon systems and software，audio alarm

• Unified management of national and regional resources

• Real-time monitoring and historical statistical analysis of failure

• Automatic reporting of system availability & statistical analysis of failure

• Fault handling workflow & fault knowledge database

• Model job monitoring

• Real-time monitoring of memory, CPU utilization and jobs

Operation monitoring system

• Code Management System -Perforce(since 2010)

• P4 using on IBM and PI-Sugon HPC

• GRAPES_GFS,GRAPES_MESO,BCC_CSM code repository

• National & regional distributed design for GRAPES_MESO collaboration

• Code version control and integration control

• Git-based Code Management System-METCODE（since 2019）

• Github-liked web remote repository and local repository，easier to share.

METCODE

• GRAPES Integrated Setting Experiment Tool

• Experiment construction

• Experiment scheduling (ecFlow)

• Experiment sharing, statistics, compare

• Integrated code and experiment data

management

• Design and implementation based

on C/S mode

• Coded by python

• Back-end services run on servers

GRAPES Integrated Setting Experiment Tool(GISET)

Programming Model on TaihuLight

Development on Sunway TaihuLight

Efforts are made to exploit many-core acceleration technology

Outline

• High Performance Computing at CMA

• Current Status of CMIP6 at CMA

19

State Council

Headquarters

Office

Department ofIntegrated

Observations.

Department of EmergencyResponse,Disaster MitigationandPublic Services

Department of Science & Technology and Climate Change

Department of Human Resources

Department of Planning&Finance

Department of Policy &Regulations

Department of InternationalCooperation

National Met.Centre/NWP Centre

National

Sat. Met. Centre

Chinese

AcademyofMet. Sciences

CMA

Training

Centre

Met.press

Provincial Met. Services 31

China Met.Society

Specialized Research Institutions

Under CMA (8)

Prefecture Met. Offices 329

County Weather Offices 2155

ChinaMet.Newspaper

CMA

National ClimateCentre

NationalMet.InfoCentre

CMAMet. Observation Centre

Department of Forecasting and Networking

WMO Regional Climate Centre

CMA Climate Change Centre

Public Service Centre

CMA Communication and Outreach Centre

CMA Asset Operation Centre

Regional Climate Centres

BCC in CMA

BCC Organization

National Climate Center (NCC)

Beijing Climate Center (BCC)

General Office (GO)

6 FTE

Division of Operation, Science and Technology

(DOST)

7 FTE

Division of Personnel Affairs (DPA)

4 FTE

Climate System Modeling Division (CSMD)

27FTE

Climate Services Division (CSD)

16 FTE

Laboratory for Climate Studies (LCS)

22 FTE

Climate Prediction Division (CPD)

31 FTE

Meteor. Disaster Risk Management Division

(MDRMD)

20 FTE

Climate Change Division (CCD)

24 FTE

Operational System Management Division (OSMD)

15FTE

FTE：Full Time Employee

Total :196Climate Application

8 FTE

Smart clouds(Hardware)

Climate data & products center

CIPAS(climate monitoring and prediction system)

Climate disaster risk management system

Climate application service platform(CLAP)

APP Web(NCC)Blog/Wechat

Op

era

tion

Sta

nd

ard

s

Tech

no

log

y fra

mew

ork

Climate modelsystems

Climate operational simulation platform

Overview of Climate Operational Systems

BCC-ESM1.0 BCC-CSM2-MR BCC-CSM2-HR

BCC_AGCM3(T42L26,Top:2.19 hPa)Top: 2.19 hPaBCC_AVIM2.0(T42)MOM4-L40v2(1/3~1°)SIS(1/3~1°)BCC-AGCM3-Chem

BCC_AGCM3(T106L46, Top:1.46hPa)BCC_AVIM2.0(T106)MOM4-L40v2(1/3~1°)SIS(1/3~1°)

BCC_AGCM3(T266L56: Top:0.1hPa)BCC_AVIM1.0(T266)MOM4-L40v2 (1/3~1°)SIS(1/3~1°)

middle resolution (110km)low resolution (280km) high resolution (45km)

Climate models developed by BCC

CMIP6: 21

About 6 staffs get involved in NCC

Computing and storage resource

roughly 50,000,000 core hours and200TB

CMIP6 Mips being carried out with BCC-CSM2-MR

Short name of MIP Model version Status

1 DECK BCC-ESM1 and BCC-CSM2-MRFinished

historicalBCC-ESM1 and BCC-CSM2-MR Finished

2 ScenarioMIP BCC-CSM2-MR Finished Tier1

3 C4MIP BCC-CSM2-MR Finished Tier1

4 CFMIP BCC-CSM2-MR Finished Tier1

5 DAMIP BCC-CSM2-MR Finished Tier1

6 GMMIP BCC-CSM2-MR Finished Tier1

7 LS3MIPBCC-AVIM2,

BCC-CSM2-MRFinished Tier1

8 LUMIP BCC-CSM2-MR Finished Tier1

9 DCPP BCC-CSM2-MR Ongoing

10AerChemMIP BCC-ESM1

Finished Tier1

11 HighResMIP BCC-CSM2-HR Ongoing

Tier 1 experiments of MIPS will be carried out.

BCC_AGCM2 ---> CMIP5 BCC_AGCM3 --- > CMIP6

Originated from CAM3

A component of BCC-CSM1.1, BCC-CSM1.1m

Resolution: T42L26, T106L26

Model Dynamic Core:

A modified dynamic framework for atmospheric spectral model (Wu et al., 2008: J.Atmos.Sci.)

Model Physics

Deep convection：Wu T., 2012: A Mass-Flux Cumulus Parameterization Scheme for Large-scale Models: Description and Test with Observations, Clim. Dyn., 38

Dry Adiabatic adjust scheme

Snow cover fraction parameterization (Wu T. and Wu G., 2004)

A modified sensible and latent flux parameterization on the ocean- Atmosphere interface (Wu et al. 2010: Climate Dynamics)

No indirect effects of aerosols

Ref: Wu et al. 2010: Climate Dynamics

A component of BCC-CSM2-MR, BCC-CSM2-HR

Resolution: T106L46, T266L56

Model Dynamic Core:

The spatially variant divergence damping scheme in higher resolution version (Whitehead et al., 2011)

Model Physics:

A gravity wave drag generated by orography, and convection (Beres et al., 2004)

A modified Wu’2012 deep convective scheme

A new scheme to parameterize deep and shallow cumulus cloud amount.

A modified parameterization scheme for surface turbulent fluxes between air and ocean/sea ice

Indirect effects of aerosols. The liquid cloud droplet number concentration is diagnosed using the aerosols mass

AGCM component

Time series of global (60°S to 60°N) mean surface air temperature from 1850 to 2015. The reference climate is for the period of 1961 to 1990.

Historical simulation of global SAT and precipitation

Summer mean precipitation climatology during 1980-2005.

BCC-CSM2-MR

BCC-CSM1.1m

GPCP

Summer precipitation in China

BCC-CSM1.1m 320x160(1.125x1.12)CCSM4 288x192 (1.25x0.95)CESM1(BGC) 288x192 (1.25x0.95)CESM1(CAM5) 288x192 (1.25x0.95)CMCC-CM 480x240(0.75x0.75)CNRM-CM5 256x128 (1.4x 1.4)MIROC4h 640x320(0.5625x0.56)MIROC5 256x128(1.4x1.4)MRI-ESM1 320x160(1.125x1.12)MRI-CGCM3 320x160(1.125x1.12)

The rainfall amount in southeast China is larger than BCC-CSM1.1m, closer to OBS than BCC-CSM1.1m.

BCC-CSM-MR can well simulate the distributions of East China summer precipitation. The spatial correlation is among the top four models.

BCC-CSM2-MR BCC-CSM1.1m

CMIP5 MME OBS

CMIP5 models with resolution higher than 1.5°

East Asian Summer Monsoon

850hPa winds (vectors) and U200 (shaded).

East Asian Monsoon Skill

BCC-CSM2-MR

BCC-CSM1.1m

CMIP5 MME9

JRA55

BCC-CSM-MR improves the skill in the simulation of East Asian summer monsoon relative to BCC-CSM1.1m.

The skill of BCC-CSM2-MR is among the top four models.

Projection of temperature and precipitation change in 21 century over China and US

with BCC-CSM2-MR in SSPs

Relative to the 20-yr mean of piControl

Climate sensitivity of BCC-CSM2-MR

Projection of global mean temperature and precipitation

Global mean SAT

Global mean land precipitation

Warming in long-term (2081-2100)

SSP585: 3.3℃SSP370: 3.0℃SSP245: 1.8℃SSP126: 0.9℃

Rainfall change in long-term(2081-2100)

SSP585: 0.10 mm/daySSP370: 0.09 mm/daySSP245: 0.06 mm/daySSP126: 0.03 mm/day

One of the early online Models

https://pcmdi.llnl.gov/CMIP6/ArchiveStatistics/esgf_data_holdings/

CMIP6 status: data availability

Check status at PCMDI website below

Online product of BCC models