26 ACTA METEOROLOGICA SINICA VOL.26

Preliminary Comparison of the CMA, ECMWF, NCEP, and JMAEnsemble Prediction Systems

DUAN Mingkeng∗(���), MA Juhui (���), and WANG Panxing (���)

Key Laboratory of Meteorological Disaster of Ministry of Education, Nanjing University of

Information Science & Technology, Nanjing 210044

(Received November 23, 2010; in final form November 29, 2011)

ABSTRACT

Based on The Observing System Research and Predictability Experiment (THORPEX) Interactive GrandGlobal Ensemble (TIGGE) dataset, using various verification methods, the performances of four typical en-semble prediction systems (EPSs) from the China Meteorological Administration (CMA), the EuropeanCentre for Medium-Range Weather Forecasts (ECMWF), the US National Centers for Environmental Pre-diction (NCEP), and the Japan Meteorological Agency (JMA) are compared preliminarily. The verificationfocuses on the 500-hPa geopotential height forecast fields in the mid- and high-latitude Eurasian regionduring July 2007 and January 2008. The results show that for the forecast of 500-hPa geopotential height,in both summer and winter, the ECMWF EPS exhibits the highest forecast skill, followed by that of NCEP,then by JMA, and the CMA EPS gets in the last. The better system behaviors benefit from the better com-bination of the following: data assimilation system, numerical models, initial perturbations, and stochasticmodel perturbations. For the medium-range forecast, the ensemble forecasting can effectively filter out theforecast errors associated with the initial uncertainty, and the reliability and resolution (the two basic attri-butions of the forecast system) of these EPSs are better in winter than in summer. Specifically, the CMAEPS has certain advantage on the reliability of ensemble probabilistic forecasts. The forecasts are easy to beunderestimated by the JMA EPS. The deficiency of ensemble spread, which is the universal problem of EPS,also turns up in this study. Although the systems of ECMWF, NCEP, and JMA have more ensemble mem-bers, this problem cannot be ignored. This preliminary comparison helps to further recognize the predictioncapability of the four EPSs over the Eurasian region, provides important references for wide applications ofthe TIGGE dataset, and supplies useful information for improving the CMA EPS.

Key words: TIGGE, ensemble prediction system, comparison, verification

Citation: Duan Mingkeng, Ma Juhui, and Wang Panxing, 2012: Preliminary comparison of the CMA,ECMWF, NCEP, and JMA ensemble prediction systems. Acta Meteor. Sinica, 26(1), 26–40,doi: 10.1007/s13351-012-0103-6.

1. Introduction

As the atmosphere is a typical chaotic sys-

tem, small initial errors in a weather forecast model

may grow rapidly to affect the model’s predictability

(Lorenz, 1963). Based on this, ensemble forecasting

was proposed as being the integration of multiple fore-

casts from slightly different initial states to provide an

estimate of the probability density function (PDF) of

forecasts (Leith, 1974). The ensemble approach was

first operationally employed by the European Cen-

tre for Medium-Range Weather Forecasts (ECMWF)

(Buizza and Palmer, 1995; Molteni et al., 1996) and

the US National Centers for Environmental Predic-

tion (NCEP, previously the National Meteorological

Center) (Tracton and Kalnay, 1993; Toth and Kalnay,

1993) in 1992, respectively, with different initial per-

turbation generation methods. Thereafter, a lot of

countries, such as Canada, Japan, China, etc., estab-

lished their own ensemble prediction systems (EPSs)

and put them into operation in different ways, includ-

ing different initial perturbation generations, multiple

forecast models, varied model resolutions, a variety

of model physics perturbations, changed number of

Supported by the China Meteorological Administration Public Welfare Research Fund (GYHY200706001 and GYHY200906007)and the Priority Academic Development Project of Jiangsu Higher Education Institutions.

∗Corresponding author: mingkeng@nuist.edu.cn.

©The Chinese Meteorological Society and Springer-Verlag Berlin Heidelberg 2012

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 27

ensemble members, and so on (Houtekamer, 1996;Park et al., 2008). For different EPSs, skill assess-ment and comparison are conducted to explore causesfor the difference in forecast skills and to determinethe way to improve the forecast system.

Due to limited forecast datasets, earlier EPS per-formance comparison studies were usually carried outbetween single deterministic forecast and EPS, or be-tween EPSs of NCEP and ECMWF (Atger, 1999;Froude et al., 2007; Mullen and Buizza, 2001; Wei andToth, 2003). The results show that the ensemble fore-casting is significantly better than single deterministicforecasting and the skill difference between the EPSsof NCEP and ECMWF is not significant. Bourke et al.(2004) compared the skills of ECMWF and BoM (Bu-reau of Meteorology of Australia) EPSs in the South-ern Hemisphere. Their results show that the ECMWFEPS is overall better than the BoM EPS. More com-prehensive comparison was analyzed by Buizza et al.(2005) among the NCEP, ECMWF, and CMC (Cana-dian Meteorological Centre) EPSs. They found thatthose EPSs all have their own advantages, but theECMWF EPS has the highest forecast skill, which isattributed to not only the perturbation method, butalso the superior model and data assimilation system.

In order to accelerate improvements in the ac-curacy of 1-day to 2-week high-impact weather fore-casts, the World Meteorological Organization (WMO)organized The Observing System Research and Pre-dictability Experiment (THORPEX) in 2003 (Shapiroand Thorpe, 2004). As a key component of THOR-PEX, TIGGE (THORPEX Interactive Grand GlobalEnsemble) project was implemented in 2005. One ofTIGGE’s objectives is to enhance international col-laboration on development of ensemble prediction be-tween operational centers and universities (Richard-son et al., 2005). Since February 2008, three TIGGEdata archive centers were established at the ECMWF,NCEP, and CMA (China Meteorological Administra-tion) to deliver the near real-time forecasting datafrom ten EPSs. From then on, with the increasingamount of data from TIGGE, it becomes easier forthe researchers to access the EPS dataset worldwide.Bougeault et al. (2010) gave a comprehensive descrip-

tion about TIGGE and performed case studies withthe EPS dataset. Park et al. (2008) compared eightEPSs and found that the best and worst control andensemble mean forecasts had large differences of about2 days of predictability for 5-day forecasts. The com-parison between the ECMWF and UKMO (UnitedKingdom Met Office) EPSs shows that the ECMWFEPS has higher skill than the UKMO EPS (Titley etal., 2008). Froude et al. (2007) and Froude (2010)analyzed the prediction of Northern Hemisphere ex-tratropical cyclones by different EPSs. Their resultsshow that the ECMWF ensemble mean and controlforecasts have the highest skill for all cyclone proper-ties.

In China, many ensemble forecasting studies havealso been carried out (Gong et al., 1999; Jin et al.,2002; Chen et al., 2005; Duan and Wang, 2006; Muand Jiang, 2008; Ma et al., 2009). Most of the EPSperformance studies were limited in assessment of oneEPS’s capability or in comparison between the deter-ministic forecast and the ensemble forecast (Huangfu,2002; Zhou et al., 2006; Wang et al., 2007). Duan(2006) and Duan et al. (2009) introduced the NCEPEPS dataset into the verification study of general cir-culation forecasting in a Eurasian region. These stud-ies prove that it is feasible to use the NCEP EPSto forecast the medium-range synoptic processes inChina, and the EPS has a higher skill in forecasts ofthe western Pacific subtropical high and the Eurasianmiddle-high latitude blocking than the deterministicforecasts. With establishment of the TIGGE archivecenter at the CMA, a good opportunity has come topromote the global and regional ensemble technologydevelopment and application in China. This helps notonly to recognize the advantages and shortcomings ofthe various operational EPSs all over the world for up-grade of the current global EPS, but also to build thesuper EPS which is suitable to the synoptic and cli-matic features of China. Before the TIGGE dataset isfully used in these ways, a necessary step is to give acomprehensive verification on the various EPSs. Suchan evaluation can provide references for users whenprocessing the forecast products from those systems.

The aim of this paper is to analyze and compare

28 ACTA METEOROLOGICA SINICA VOL.26

the predictions of 500-hPa geopotential height (Z500)in the Eurasian region by four different EPSs archivedin the TIGGE dataset. This topic is consistent withthe other key objectives of TIGGE, which are to de-velop new methods to combine ensembles of predic-tions from different sources and to correct systematicerrors. This paper is perhaps particularly in time toserve as the basis of medium-range ensemble forecast-ing in the Eurasian region, especially in China.

The data used in this study are described in Sec-tion 2, followed by a description of the analysis meth-ods in Section 3. As the main part, the skills of en-semble mean forecasts and probabilistic forecasts arecompared respectively in Sections 4 and 5. Section 6presents the conclusions and discussion.

2. Data description

This study analyzes the four EPSs from the CMA,ECMWF, NCEP, and JMA (the Japan Meteorologi-cal Agency), respectively, in the Eurasian region (15◦–

80◦N, 40◦–150◦E) during July 2007 (31 days, here-inafter referred to as Period I) and January 2008(31 days, as Period II). The two periods can be re-garded as the representatives of summer and winter,respectively. Table 1 shows the main characteristics ofthese EPSs. For the initial perturbation methods, theECMWF EPS uses singular vectors (SVs), the NCEPand CMA systems use bred vectors (BVs), while theJMA system uses the two methods during the two pe-riods with the upgrade of model resolution. Differentfrom the other three centers, ECMWF applies randomperturbations to the numerical model (Buizza et al.,1999). The compared variable here is Z500 on hor-izontal grids of 2.5◦ by 2.5◦. The other informationabout the forecast data used here is shown in Table 2.Forecasts are verified every 6 h from the lead time of 6h to the maximum. Because of transmission problems,except the CMA EPS, the samples of the other threesystems are missing more or less. Especially, the JMAEPS has only nine samples in Period I. Therefore, itscredibility of the corresponding verification results is

Table 1. General profiles of the four TIGGE ensembles used in this studyForecast system CMA EPS ECMWF EPS JMA EPS NCEP EPS

Initial perturbation BVs SVs BVs (before November 2007) BVs

generation method SVs (from November 2007)

Model uncertainty No Yes No No

TL399 (0–10 days) TL159 (before November 2007)

Horizontal resolution T213 TL255 (10–15 days) TL319 (from November 2007) T126

Number of vertical levels 31 62 40/60 28

Maximum lead time (day) 10 15 9 16

Start time every day 00Z, 12Z 00Z, 12Z 12Z 00Z, 06Z, 12Z, 18Z

Number of perturbation 14 50 50 20

members

Table 2. Overview of the ensemble forecasting data in this studyForecast system CMA EPS ECMWF EPS JMA EPS NCEP EPS

Start time every day 00Z 00Z 12Z 00Z

Number of samples/days (Period I) 31 25 9 28

Number of samples/days (Period II) 31 13 25 12

Forecast lead time (day) 0.25–10 0.25–15 0.25–9 0.25–16

Number of perturbation members 14 50 50 20

relatively lower.Besides the TIGGE data, this study also uses the

NCEP/NCAR reanalysis dataset (Kalnay et al., 1996;Kistler et al., 2001). The daily reanalysis data areused to determine 10 climatologically equally proba-ble intervals of Z500 (Toth et al., 2001) at each gridand verification time as the thresholds for the prob-

abilistic forecasts evaluation. The temporal cover-age is from January 1978 to December 2007. The4-time daily data are employed as the analysis fieldto verify the four EPSs’ performance. The vari-able, spatial and temporal coverage, and horizontalgrid of analysis data are all the same as the forecastdata.

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 29

3. Verification methods

It is well known that ensemble forecasting can in-crease the deterministic forecast skill by the ensemblemean and it uses the probability to measure the fore-cast uncertainty. Verifications of both deterministicand probabilistic forecasts are carried out here. Theverification of the deterministic forecast is relativelysimple. It is the measure of the difference betweenforecasts and observations (verifying analysis). Calcu-lations of the anomaly correlation coefficient (ACC)and root mean square error (RMSE), as the standardmethods recommended by the WMO (Liu and Zhang,1992), are adopted here.

The probabilistic forecast verification is muchmore complicated. Reliability and resolution are thetwo main attributors of forecast systems (Toth et al.,2006). Reliability is defined as the statistical consis-tency of forecasts with observations. For the station-ary process, reliability can be statistically correctedby post-process, so that the forecast consistency canbe significantly improved. Resolution is defined as theability of forecasts to distinguish among different out-comes. Different from reliability, resolution cannot bestatistically corrected. Obviously, an excellent forecastsystem should have good resolution based on good re-liability.

For the reliability evaluation of probabilistic fore-casts, rank histogram and reliability diagram are usedhere. Reliability diagram can also be employed toevaluate the resolution of probabilistic forecasts, alongwith the methods of relative operating characteris-tics (ROC) and relative economic value (EV) analysis(Richardson, 2000; Zhu et al., 2002).

Rank histogram, also known as Talagrand dia-

gram, is introduced to display bias and dispersionwithin an ensemble to verify whether the spread ofan ensemble is able to encompass the verifying analy-sis (Talagrand et al., 1997). The process to generaterank histogram is as follows. Firstly, check where theverifying analysis falls within ensemble forecasts. Forexample, the CMA EPS has 14 members; we rank theforecast values of the 14 members from the smallestto the largest, so 15 bins in total can be created, thenthe bins that fall within the verifying analysis can befound. Secondly, do the same for all the grid points,and then determine the average frequencies at whichthe verifying analysis falls into each of the bins. Fi-nally, draw these average frequencies of each of thebins in form of histogram and obtain a rank histogram(Hamill, 2001).

The reliability diagram measures the ability ofthe system to forecast accurate probabilities. Forexample, for the 20% probability forecasts, the pre-dicted event should be verified as 20%, not more orless. This diagram can be obtained by plotting valuesof forecast probability against observation frequency(Hamill, 1997).

For users of the forecasts, the failure to forecasta serious synoptic event that has occurred will havemuch more dramatic consequences than forecastingan event that does not occur. To assess the forecastskill under these conditions, other verification meth-ods should be used. For the probabilistic forecast, if athreshold is given, a forecast can be transformed intoa yes/no statement (categorical forecast) easily. Theobservation itself can be put in one of two categories(observed/not observed). Then, a frequency contin-gency (Table 3) can be obtained. In Table 3, h

denotes hit frequency, i.e., the ratio of all correct yes-

Table 3. Contingency table indicating the costs and losses accrued by the use of weather forecasts, depending on

forecast and observed events (adopted from Zhu et al. (2002))

Forecast/action

Yes No

YesHit (h) Miss (m)

Mitigated loss (C+Lu) Loss (L = Lp + Lu)Observation

NoFalse alarm (f) Correct rejection (c)

Cost (C) No cost (N)

30 ACTA METEOROLOGICA SINICA VOL.26

forecasts (the event is predicted to occur and it doesoccur) to all the forecasts; f is false alarm frequency,i.e., all incorrect yes-forecasts; m is missed forecast (allincorrect no-forecasts) frequency, and c is all correctno-forecast frequency. Obviously, h + f + m + c = 1,and o = h + m is the climatological frequency of theevent.

According to Table 3, the measure that examinesthe hits of a certain event is hit rate:

H =h

h + m=

h

o. (1)

While maximizing the number of hits (minimizing thenumber of false alarms) is desirable, it is required toexamine hit rate together with false alarm rate:

F =f

f + c=

f

1 − o. (2)

Then, a ROC curve can be obtained by plotting the hitrate (y-axis) versus the false alarm rate (x-axis). Fordeterministic forecasts, there is only one point (H, F )on the ROC curve in addition to the corner points (0,0) and (1, 1). The closer the F and H are to the up-per left corner (lower and higher), the higher the skillis. A perfect forecast system would let all its pointson the top left corner (H = 100% and F = 0). Forprobabilistic forecasts, at each probability threshold,a point (H, F ) can be obtained, so the ROC curve canbe formed with all these points as well as (0, 0) and(1, 1).

Actually, the benefits gained by using ensembleforecasting depend on not only the skill but also theactions the users take to mitigate the consequences.A decision maker has a number of courses of actionto choose from, and the choice is to some extent in-fluenced by the forecasts (Richardson, 2000). For thisreason, during forecast verification, it is necessary tointroduce the method of economic value. This methodcan comprehensively access the influence of the prob-ability forecast for various types of users, therebyachieving an overall understanding on the forecastingsystems. Also as shown in Table 3, each action has anassociated cost/loss and leads to an economic benefitdepending on the weather outcome. As described byZhu et al. (2002), if the event does occur and the useris not protected (miss), it will suffer a loss L, whichcan be denoted as the sum of the loss that can be pro-tected against (Lp), and the unprotectable loss (Lu).

If a user takes action to guard against this potentialloss, the user will incur a cost and C < L. If theevent does not occur (false alarm), C is the user’s to-tal cost; if the event occurs (hit), besides C, the usermay also incur the unprotectable loss Lu. The termC + Lu is usually called mitigated loss, and typicallyC < C + Lu < L. If the user does not take action andthe event does not occur (correct rejection), there is nocost (N = 0). Based on this, a cost-loss ratio decisionmodel can be formed, and the task is to choose theappropriate action which will minimize the expectedloss on the long term.

If only the information about the climatologicalfrequency of occurrence of the event (o) is available,the strategy reduces to either always protect or never.In this situation, the corresponding average expenseis:

Eclimate = Min[C + oLu, o(Lp + Lu)]

= oLu + Min(oLp, C). (3)

If there are perfect forecasts, the user only takes ac-tions on those occasions that the event is going to oc-cur, and the average expense would be:

Eperfect = o(C + Lu). (4)

The actual forecasts allow adapting the strategy in or-der to reduce the average expense. The reduction ofexpense is a measure of the value of the forecasts. Therelative economic value that compares the reduction ofexpense with the reduction which would be achievedby a perfect forecast, is the ratio:

V =Eforecast − Eclimate

Eperfect − Eclimate. (5)

According to Table 3, the average expense of thedeterministic forecast system is given with:

Eforecast = h(C + Lu) + fC + m(Lp + Lu)

= oLu + (h + f)C + mLp. (6)

Substituting the expenses given in Eqs. (3), (4), and(6) into the EV definition in Eq. (5) will obtain:

V =Min[o, γ] − (h + f)γ − m

Min[o, γ] − oγ, (7)

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 31

where γ = C/Lp is the cost-loss ratio, which representsthe different situations of the user. Furthermore, withthe rate of hit and false alarm in Eqs. (1) and (2), theEV of the deterministic forecast can be rewritten as:

V =Min(o, γ) − Fγ(1 − o) + Ho(1 − γ) − o

Min(o, γ) − oγ. (8)

From Eq. (8), for a certain weather event (o isfixed), V not only is related to ROC (H and F ) whichquantifies the skill of the forecast system, but alsotakes the influence of user’s situation (γ) into account.It is easy to prove that EV reaches a maximum when γ

= o. For the Z500 verification here, because it is basedon the 10 climatologically equally probable intervals,the climatological probability o = 0.1.

As far as probabilistic forecast is concerned, theuser has to choose a proper probability threshold whenan action needs to be taken. By this choice, the prob-abilistic forecast can be changed into a deterministicone. For any value of probability threshold, the rela-tive EV of the probabilistic system can be calculated

with Eq. (8). As an optimal strategy, the decisionmaker will select the probability threshold which re-sults in the largest value.

By now, the verification methods have beenbriefly introduced. It should be noticed that all theverifications above are accumulated and averaged overall the verified grid points and temporal coverage.The threshold used by probabilistic forecast evaluationhere is based on the 10 climatologically equally proba-ble intervals, which is derived from the NCEP/NCARdaily reanalysis dataset (Toth et al., 2001).

4. Comparison of ensemble mean forecasts

According to the comparison of ACCs of the fourEPSs in the two periods (Figs. 1a and 1b), the fore-cast skill in winter (Period II) is better than in summer(Period I). Benefited from the better model perturba-tion method and data assimilation system, ACC of theECMWF EPS is always the highest, followed by

Fig. 1. Ensemble mean forecast verification of Z500 in the Eurasian region. (a) and (c) are ACC and RMSE in Period

I, respectively; and (b) and (d) are the same but in Period II.

32 ACTA METEOROLOGICA SINICA VOL.26

Table 4. The maximal lead time (day) of the “useful forecasts” of Z500 in the Eurasian region during the two

periods

Forecast system CMA EPS ECMWF EPS NCEP EPS JMA EPS

Period I 6.25 8 6.5 6.75

Period II 8 13 12.75 8.5

NCEP, JMA, and CMA EPSs successively. Generally,the forecasts can be considered as the “useful fore-casts”. According to Fig. 1 and Table 4, the compari-son on the maximum lead time of the useful forecastsshows that the difference between EPSs of ECMWFand CMA is very marked, about 1.75 days (5 days)in Period I (II). The RMSE skill ranking of the fourEPSs is the same as ACC. Different from ACC, theRMSE in winter is not better than in summer.

5. Comparison of ensemble probabilistic fore-casts

5.1 Rank histogram

According to the principle of ensemble forecast-ing, for a perfect EPS, the ensemble members shouldbe uniformly sampled in the initial or the forecastedPDF. Statistically, the appearance of each ensemblemember in the PDF is equiprobable. This character-istic can be demonstrated in the rank histogram as thesituation that the verifying analysis value falls in thesebins with equal probability as discussed in Section 3.But this is just an ideal situation. The current opera-tional EPS systems do not achieve this state yet.

As shown in Figs. 2a and 2b, the rank histogramsof CMA EPS during the two periods are basicallyconsistent. At the 1-day lead time, the distributionlike the letter “W” shows that the verifying analysisis more likely to fall in the middle of the ensemblesampling categories or to fall outside the categories,which possibly means that the spread of ensemble isunstable, always easy to be too large or too small.With the increasing lead time, the distributions areevolved into U-shape (the 5-, 7-, and 9-day panels inFigs. 2a and 2b), which is an indication of a positivebias in the variance of the ensemble. Generally, for theCMA EPS, the distributions have little difference, andbecome consistent with the growing lead time. Thisshould be associated with the fewer ensemble mem-

bers, as well as the smaller ensemble spread.For the ECMWF EPS (Figs. 2c and 2d), when

the lead time is short, the rank histogram distribu-tion is U-shaped, which corresponds to the inadequateensemble spread. The situation is slightly better insummer than in winter. Benefited from the systemsuperiority, with the increasing lead time, the frequen-cies in different bins become equal. Different from theECMWF EPS, the NCEP EPS always has a lower en-semble spread throughout the whole lead time andin different seasons (Figs. 2e and 2f). This meansthat the NCEP EPS has more stability but less self-adjusting capability. The distribution of the JMA EPSis asymmetrically U-shaped, more like an inverted L-shape, which represents a bias in the mean of the fore-casts (Figs. 2g and 2h). In addition, the reliability ofthe JMA EPS in summer is also obviously better thanin winter as the ECMWF EPS.

From the above analysis, it can be found that thecurrent main operational EPS systems over the worldalways have the U-shape rank histogram, which im-plies that the verifying analysis falls outside the cloudof ensemble forecasts more often than one can expectby chance, given the finite ensemble size. In otherwords, the current EPS still underestimates the trueuncertainty in the forecasts.

5.2 Reliability diagram

Based on the climatologically equally probable in-tervals, Fig. 3 shows the reliability diagram of Z500ensemble probabilistic forecasts. According to the def-inition, for a perfectly reliable forecast system, theforecast probabilities should be equal to the observedfrequencies. For the diagram as in Fig. 3, the corre-sponding reliability curve should lie along the diago-nal.

In Fig. 3, it can be found that the 1-day reliabil-ity is not better than 3- and 5-day lead time, becausethere is an adjusting process similar to the dynamic

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 33

Fig. 2. Rank histograms for Z500 ensemble forecasts in the Eurasian region. (a), (c), (e), and (g) are the results for theCMA, ECMWF, NCEP, and JMA EPSs in Period I, respectively; and (b), (d), (f) and (h) are the same but for PeriodII.

34 ACTA METEOROLOGICA SINICA VOL.26

initialization of numerical weather prediction at thebeginning of the forecast. Beyond the 5-day lead time,the reliability moves away from the diagonal gradu-ally, but the case in Period II is better than in PeriodI. In Period I, for the event whose forecast probabilityis more than 60%, its observed frequency is signifi-cantly lower. For example, the CMA EPS forecastprobability is about 90%, but the actual observationfrequency is only about 50% in Period I, and reachesabout 80% in Period II. Except for the 1-day lead time,the ECMWF EPS always has the highest reliability,while the NCEP EPS has the lowest. As discussed inSection 3, the forecast reliability can be greatly cal-ibrated to be in accord with the observed frequencywith the help of historical data. Just for this reason,despite that the NCEP EPS is not reliable enough ini-tially, it still has a good chance of getting good relia-bility by the statistical calibration. It should be notedthat the deviation from diagonal is not necessarily in-dicative of the true deviation of reliability; this maybe due to sampling variations. Thereafter, with thepoor samples here, the JMA EPS has the unstable re-liability, especially in the 9-day forecasts in Period I(Fig. 3a).

In addition to the reliability measurement, thereliability diagram can also be used to measure theresolution range of probability forecasts. When theforecast frequency grows from 0% to 100% along thereliability curve, the corresponding range of the ob-served frequency is the range of the forecast resolu-tion. For instance, the resolution range of 9-day fore-casts of the CMA EPS in Period I is about 5%–70%(Fig. 3a). As discussed above, the resolution attri-bution of probabilistic forecasts cannot be calibrated,which means that in the reliability diagram, the pointcan only be calibrated to move along the horizontaldirection to the diagonal. For the uncalibrated relia-bility curve, the reliability and resolution ranges areassociated. The closer the curve gets to the diagonal,the better system reliability and the wider resolutionrange correspond. Therefore, the ECMWF EPS al-ways has the highest resolution range, while the otherthree systems have comparable resolution ranges.

5.3 ROC analysis

As shown in Fig. 4, the ROC curve demonstratesthe hit rate and the false alarm rate of the proba-bilistic prediction on different probabilistic thresholds.The probabilistic forecasts can be converted into de-terministic forecasts given that the probability exceedsa certain threshold. Every point on the curve fromlower left to upper right corresponds to the thresh-old from 0% to 100%, respectively. The closer thecurve is to the upper left corner, the higher the hitrate and the lower the false alarm correspond, and thebetter forecasts are obtained. In Fig. 4, on the lowprobability threshold, the false alarm rate in summeris higher than in winter. At 1-day lead time, exceptthe CMA EPS, the other three systems have roughlyequal resolutions in Period I, while the JMA systemhas the worse resolution in Period II. As the lead timegoes longer, the curve is closer to the diagonal, whichmeans that there is a decline of the forecast skill. Atthe same time, the difference of the four systems isreduced. In Period I, the resolution of the ECMWFEPS is always the highest, the CMA EPS is the low-est, and the other two are in the middle. The skill ofthe NCEP EPS is slightly better than the JMA EPSin winter.

Based on the ROC curve in Fig. 4, Fig. 5 showsthe corresponding ROC area, which is the area un-der the ROC curve. The larger area corresponds tothe higher hit rate and the lower false alarm rate, i.e.,the better forecast skill. From Fig. 5a, the area ofthe ECMWF system is always the biggest in summer.When the lead time is less than 6 days, the capabilitiesof the NCEP and JMA EPSs are comparative. Afterthat, the ROC area of the JMA EPS begins to growgradually and is close to the level of ECMWF, whilethe NCEP EPS area steadily sinks down. For all thelead times, the area of the CMA EPS is the smallest.In winter (Fig. 5b), when the lead time is less than 1day, the areas of the NCEP and CMA EPSs are largerthan those of the JMA and ECMWF. After the 2-daylead time, the ECMWF EPS is the biggest, followedby JMA, while the areas of the NCEP and CMA EPSs

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 35

are the smallest. Until the 8-day lead time, the area

of the NCEP EPS begins to expand and is close to

the ECMWF system. If the system performance of

the earlier 9-day forecasts is compared between the

two periods, it can be found that the forecast skill

is in substantial agreement. After that, the skills of

the ECMWF and NCEP EPSs in Period II are better

than in Period I.

Fig. 3. Reliability diagrams of Z500 ensemble probabilistic forecasts in the Eurasian region for (a) Period I and (b)

Period II.

36 ACTA METEOROLOGICA SINICA VOL.26

Fig. 4. ROC curves of Z500 ensemble probabilistic forecasts in the Eurasian region for (a) Period I and (b) Period II.

5.4 EV analysis

Based on the ROC analysis, the EV analysis com-

bines the results of probability prediction with the

decision-making of the specific forecast users. Accord-

ing to Section 3, EV is the relative measure to the

climate status (V = 0). V > 0 means that the fore-

casts are valuable and users can benefit from the sys-

tem, and vice versa. In other words, whether the users

have benefited from the system can be measured by

the range of the cost/loss ratio γ with V > 0.

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 37

Fig. 5. ROC area of Z500 ensemble probabilistic forecasts in the Eurasian region for (a) Period I and (b) Period II.

Fig. 6. EV of Z500 ensemble probabilistic forecasts in the Eurasian region for Period I (left panels) and Period II (right

panels).

Figure 6 shows the relative EV of the four sys-

tems in summer and winter. For the 1-day lead time

in Period I, when γ < 0.1, except that the EV of the

CMA EPS is obviously lower, the other three systems

have comparable resolutions. However, when γ > 0.1,

the EV of the JMA EPS is the highest, followed by

38 ACTA METEOROLOGICA SINICA VOL.26

Fig. 7. The maximal EV of Z500 ensemble probabilistic forecasts in the Eurasian region for (a) Period I and (b) Period

II.

NCEP and ECMWF systems, and the EV of the CMAsystem is still the minimal. If the ECMWF EPS isused by the users of γ > 0.8, the EV will be the min-imal. For the situations shown in Fig. 6a, the usersbenefited from the ECMWF EPS are fewer than thosefrom the other three systems. In Period II, except thesituation of γ > 0.9 in 1-day lead time, the ECMWFEPS still has the highest EV for any users. Next is theJMA system, which has higher values in all the leadtime except for that of 1 day. The CMA EPS does notoffer any obvious advantage yet.

As discussed in Section 3, when γ = o, the cor-responding EV reaches the maximum. Here the max-imal EV is at the position of γ = 0.1 as shown inFig. 6. Based on the results of Fig. 6, Fig. 7 showsthe maximal EV of the four systems. It also provesthat the maximum EV is equal to the minus of the hitrate and the false alarm rate in the ROC analysis, i.e.,Vmax = H − F . Just for this reason, the result here isconsistent with the result of ROC area (Fig. 5) andthe two curves have very similar shapes. Obviously,the skill of the ECMWF system is still the highest,while the CMA system needs to be improved greatly.

6. Conclusions and discussion

Based on the TIGGE dataset, this study com-pares the Z500 forecast skills of the four typical EPSsfrom the CMA, ECMWF, NCEP, and JMA, respec-tively, in the middle- and high-latitude Eurasian re-gion during July 2007 and January 2008. The skill ver-

ifications focus on the ensemble mean and probabilisticforecasts. During the comparison, various methods,such as ACC, RMSE, rank histogram, reliability dia-gram, ROC, and EV analysis, are used. The resultscan be summarized as follows.

1) For the overall performance, whether in winteror summer, benefited from the better data assimila-tion system, the better numerical models, and bet-ter stochastic model perturbations, the skill of theECMWF EPS is the highest and followed by theNCEP, JMA, and CMA systems in order.

2) The reliability and resolution of the forecastsare better in winter than in summer. Besides theseasonal difference of atmospheric predictability, theother reason is that ensemble forecasting can effec-tively filter out the forecast errors generated by theinitial uncertainty.

3) To be specific, the CMA EPS has certain ad-vantage in the reliability of probabilistic forecasts. Allthe four systems have inadequate ensemble spread,which means that the verifying analysis cannot alwaysbe included into the sampling space of the forecastedPDF. Actually, this is a common shortcoming of theEPS at present, which is more evident in the ECMWFand NCEP systems, despite that they have more en-semble members. For the JMA system, the forecastsare easy to be underestimated.

In brief, this preliminary comparison helps to fur-ther recognize the forecast skill of the four typical sys-tems in the Eurasian region where the medium-rangesynoptic processes influencing China occur. To use the

NO.1 DUAN Mingkeng, MA Juhui and WANG Panxing 39

TIGGE dataset more extensively, the above resultswill be a very important reference. For instance,the results can be used to construct the super EPS,TIGGE-based mesoscale or microscale nested EPS,end-to-end interactive EPS, and so on. With the de-velopment over the past decade, great progress hasbeen made, but the ensemble forecasting system ofChina still has a considerable gap to the interna-tional level. In order to improve the skill of the CMAEPS, based on this study, there is still a great dealof innovative work to do, such as data assimilationsystem upgrade, numerical model (e.g., GRAPES)development, ensemble initial perturbation techniqueimprovement, and ensemble forecasting productionresearch.

As mentioned in Section 2, it should be pointedout that the number of forecast samples here is rel-atively less than expectation. Undoubtedly, this willexert a negative influence on the reliability of ourverification results. In order to confirm the above con-clusion, the differences of the forecast skill between theNCEP and ECMWF EPSs in this study are comparedwith some other conclusions of the former studies,such as Buizza et al. (2005) and Park et al. (2008).The results show that the conclusions of skill compar-ison of the two centers here are generally consistentwith those from former studies, which means that theconclusions here are reliable. From Table 2, the sam-ple numbers of the two EPSs are less than that of theCMA, thus it can be concluded that the verificationof the CMA EPS is also objective. By this way, thereliability of this study can be partly confirmed fromanother aspect.

Moreover, in addition to the preliminary studyabout the general circulation forecast skill here, in or-der to obtain comprehensive knowledge on the EPSs inTIGGE, a lot of verification studies should be carriedout for more systems (e.g., CMC, BoM, and UKMOEPSs), for different variables at different levels (e.g.,temperature at 850 hPa and 2 m, wind at 200 hPa) inthe future. This can immensely improve the EPS per-formance and exert the usage of the TIGGE datasetto the fullest extent.

Acknowledgments. We are indebted to HuJiangkai, Ma Ming, and Tian Weihong, who provided

the TIGGE dataset.

REFERENCES

Atger, F., 1999: The skill of ensemble prediction systems.

Mon. Wea. Rev., 127, 1941–1953.

Bougeault, P., Z. Toth, C. Bishop, et al., 2010: The

THORPEX interactive grand global ensemble. Bull.

Amer. Meteor. Soc., 91, 1059–1072.

Bourke, W., R. Buizza, and M. Naughton, 2004: Per-

formance of the ECMWF and the BoM ensemble

prediction systems in the Southern Hemisphere.

Mon. Wea. Rev., 132, 2338–2357.

Buizza, R., and T. N. Palmer, 1995: The singular-vector

structure of the atmospheric global circulation. J.

Atmos. Sci., 52, 1434–1456.

—–, M. Miller, and T. N. Palmer, 1999: Stochastic rep-

resentation of model uncertainties in the ECMWF

ensemble prediction system. Quart. J. Roy. Me-

teor. Soc., 125, 2887–2908.

—–, P. L. Houtekamer, Z. Toth, et al., 2005: A com-

parison of the ECMWF, MSC, and NCEP global

ensemble prediction systems. Mon. Wea. Rev.,

133, 1076–1097.

Chen Jing, Xue Jishan, and Yan Hong, 2005: A new

initial perturbation method of ensemble mesoscale

heavy rain prediction. J. Appl. Meteor. Sci., 29,

717–726. (in Chinese)

Duan Mingkeng, 2006: Ensemble forecasting performance

studies on the important medium-range circulation

processes affecting China. Ph. D. dissertation, Nan-

jing University of Information Science & Technology,

China, 144 pp. (in Chinese)

—– and Wang Panxing, 2006: A new weighted average

method on ensemble mean forecasting. J. Appl.

Meteor. Sci., 17, 488–493. (in Chinese)

—–, —–, Wu Hongbao, et al., 2009: The ensemble fore-

casting verification on the summer Eurasian middle-

high latitude circulation. J. Appl. Meteor. Sci., 20,

56–61. (in Chinese)

Froude, L. S. R., 2010: TIGGE: Comparison of the

prediction of Northern Hemisphere extratropical

cyclones by different ensemble prediction systems.

Wea. Forecasting, 25, 819–836.

—–, L. Bengtsson, and K. I. Hodges, 2007: The prediction

of extratropical storm tracks by the ECMWF and

NCEP ensemble prediction systems. Mon. Wea.

Rev., 135, 2545–2567.

Gong, J. D., W. J. Li, and J. F. Chou, 1999: Forming

proper ensemble forecast initial members with

40 ACTA METEOROLOGICA SINICA VOL.26

four-dimensional variational data assimilation

method. Chinese Sci. Bull., 44, 1527–1531.Hamill, T. M., 1997: Reliability diagrams for multicate-

gory probabilistic forecasts. Wea. Forecasting, 12,

736–741.—–, 2001: Interpretation of rank histograms for verifying

ensemble forecasts. Mon. Wea. Rev., 129, 550–560.Houtekamer, P. L., L. Lefaivre, J. Derome, et al., 1996: A

system simulation approach to ensemble prediction.

Mon. Wea. Rev., 124, 1225–1242.Huangfu Xueguan, 2002: Verification of the ensemble

prediction system of the National Meteorological

Center. J. Appl. Meteor. Sci., 13, 29–36. (in Chi-

nese)Jin Ronghua, Li Weijing, and Sun Zhaobo, 2002: Study

on the interpretation method of ensemble prediction

products. J. Appl. Meteor. Sci., 25, 442–448. (in

Chinese)Kalnay, E., M. Kanamitsu, R. Kistler, et al., 1996: The

NCEP/NCAR 40-year reanalysis project. Bull.

Amer. Meteor. Soc., 77, 437–472.Kistler, R., E. Kalnay, W. Collins, et al., 2001: The

NCEP/NCAR 50-year reanalysis: Monthly means

CD-ROM and documentation. Bull. Amer. Meteor.

Soc., 82, 247–268.Leith, C. E., 1974: Theoretical skill of Monte Carlo fore-

casts. Mon. Wea. Rev., 102, 409–418.Liu Huanzhu and Zhang Shaoqing, 1992: Statistical veri-

fication on the medium-range numerical forecasting.

Meteor. Mon., 18, 50–54. (in Chinese)Lorenz, E. N., 1963: Deterministic nonperiodic flow. J.

Atmos. Sci., 20, 130–141.Ma Xulin, Xue Jishan, and Lu Weisong, 2009: Study

on ETKF-based initial perturbation scheme for

GRAPES global ensemble prediction. Acta Me-

teor. Sinica, 23, 562–574.Molteni, F., R. Buizza, T. N. Palmer, et al., 1996: The

ECMWF ensemble prediction system: Methodology

and validation. Quart. J. Roy. Meteor. Soc., 122,

73–120.Mu, M., and Z. N. Jiang, 2008: A new approach to

the generation of initial perturbations for ensemble

prediction: Conditional nonlinear optimal perturba-

tion. Chinese Sci. Bull., 53, 2062–2068.Mullen, S. L., and R. Buizza, 2001: Quantitative pre-

cipitation forecasts over the United States by the

ECMWF ensemble prediction system. Mon. Wea.

Rev., 129, 638–663.Park, Y. Y., R. Buizza, and M. Leutbecher, 2008:

TIGGE: Preliminary reliminary results on compar-

ing and combining ensembles. Quart. J. Roy. Me-

teor. Soc., 134, 2029–2050.

Richardson, D., 2000: Skill and relative economic value

of the ECMWF ensemble prediction system. Quart.

J. Roy. Meteor. Soc., 126, 649–667.

——, R. Buizza, and R. Hagedorn, 2005: Final report

of the 1st Workshop on the THORPEX Interac-

tive Grand Global Ensemble (TIGGE), WMO/TD-

No.1273, WWRP/THORPEX, No.5, 39 pp.

Shapiro, M., and A. Thorpe, 2004: THORPEX In-

ternational Science Plan. WMO/TD-No. 1246,

WWRP/THORPEX, No. 2, 51 pp.

Talagrand, O., R. Vautard, and B. Strauss, 1997: Evalua-

tion of probabilistic prediction systems. Proceedings

of the ECMWF Workshop on Predictability, Read-

ing, United Kingdom, ECMWF, 1–25.

Titley, H., N. Savage, R. Swinbank, et al., 2008: Compari-

son between Met Office and ECMWF medium-range

ensemble forecast systems. Meteorology Research

and Development Technical Report No. 512, Met

Office, United Kingdom, 41 pp.

Tracton, M. S., and E. Kalnay, 1993: Operational en-

semble prediction at the National Meteorological

Center: Practical aspects. Wea. Forecasting, 8,

379–398.

Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at

NMC: The generation of perturbations. Bull. Amer.

Meteor. Soc., 74, 2317–2330.

—–, O. Talagrand, and Y. Zhu, 2006: The attributes of

forecast systems: A general framework for the eval-

uation and calibration of weather forecasts. Pre-

dictability of Weather and Climate, T. N. Palmer

and R. Hagedorn, Eds., Cambridge University Press,

584–595.

—–, Y. Zhu, and T. Marchok, 2001: The use of en-

sembles to identify forecasts with small and large

uncertainty. Wea. Forecasting, 16, 463–477.

Wang Chenxi, Yao Jianqun, and Liang Xudong, 2007:

The establishment and verification of the operational

ensemble forecast system for Shanghai regional pre-

cipitation. J. Appl. Meteor. Sci., 18, 173–180. (in

Chinese)

Wei, M., and Z. Toth, 2003: A new measure of ensemble

performance: Perturbation versus error correlation

analysis (PECA). Mon. Wea. Rev., 131, 1549–1565.

Zhou Bing, Zhao Cuiguang, and Zhao Shengrong, 2006:

Multi-model ensemble forecast technology with anal-

ysis and verification of the results. J. Appl. Meteor.

Sci., 17, 104–109. (in Chinese)

Zhu, Y., Z. Toth, R. Wobus, et al., 2002: The economic

value of ensemble-based weather forecasts. Bull.

Amer. Meteor. Soc., 83, 73–83.