On the quality and value of probabilistic forecasts of ... · 9th International Conference on Probabilistic MethodsApplied to PowerSystems KTH, Stockholm, Sweden-June 11-15, 2006

General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

You may not further distribute the material or use it for any profit-making activity or commercial gain

You may freely distribute the URL identifying the publication in the public portal If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

Downloaded from orbit.dtu.dk on: Feb 11, 2020

On the quality and value of probabilistic forecasts of wind generation

Pinson, Pierre; Juban, Jeremie; Kariniotakis, Georges

Published in:International Conference on Probabilistic Methods Applied to Power Systems, 2006. PMAPS 2006.

Link to article, DOI:10.1109/PMAPS.2006.360290

Publication date:2006

Document VersionPublisher's PDF, also known as Version of record

Link back to DTU Orbit

Citation (APA):Pinson, P., Juban, J., & Kariniotakis, G. (2006). On the quality and value of probabilistic forecasts of windgeneration. In International Conference on Probabilistic Methods Applied to Power Systems, 2006. PMAPS2006. IEEE. https://doi.org/10.1109/PMAPS.2006.360290

https://doi.org/10.1109/PMAPS.2006.360290

https://orbit.dtu.dk/en/publications/503c313a-4a68-4bdb-90fe-f6e5a31ae53d

https://doi.org/10.1109/PMAPS.2006.360290

9th International Conference on Probabilistic Methods Applied to Power SystemsKTH, Stockholm, Sweden - June 11-15, 2006

On the Quality and Value of Probabilistic Forecastsof Wind Generation

P. Pinson, J. Juban and G. N. Kariniotakis, Member, IEEE

(Invited Paper)

Abstract-While most of the current forecasting methods interval forecasts, which translates to quoting particular pointsprovide single estimates of future wind generation, some methods of predictive distributions of wind power (see [5]-[7] amongnow allow one to have probabilistic predictions of wind power. others). In the present paper, we leave aside the case of riskThey are often given in the form of prediction intervals or quan- . . ' . .tile forecasts. Such forecasts, since they include the uncertainty indices and concentrate on probabilistc predicthons.information, can be seen as optimal for the management or Here, our aim is to describe what may be the quality andtrading of wind generation. This paper explores the differences value of probabilistic forecasts of wind generation. Theseand relations between the quality (i.e. statistical performance) two concepts have been introduced in the meteorologicaland the operational value of these forecasts. An application is forecasting literature by Murphy [8]: the quality aspect standspresented on the use of probabilistic predictions for bidding in a for the statistical performance of the probabilistic forecasts,European electricity market. The benefits of a probabilistic view forute wth a serfomanceof diagrams, whi theof wind power forecasting are clearly demonstrated. evaluated with a set of measures and diagrams, while the

Index Terms-Wind power, probabilistic forecasting, evalua- value aspect corresponds to the increased benefits (economicaltion methods, decision-making, operational value. or not) from the use of these probabilistic forecasts in an

operational context. We base the present study on uncertaintyI. INTRODUCTION estimates provided by a method previously developed by thePODUENis the fastest-growing renewable

authors [4] (i.e. adapted resampling), which can be consideredW1iND POWER is the fastest-growing renewable for post-processing state-of-the-art wind power point predic-

electricity-generating technology. The targets for the tion methods. The test case consists in a multi-MW windnext decades aim at high share of wind power in electricity farm located in Denmark, for which probabilistic forecastsgeneration in Europe [1] (up to 75GW for 2010). However, are obtained by applying the adapted resampling approachsuch a large-scale integration of wind generation capacities in- to point predictions given by three different methods, overduces difficulties in the management of a power system. Also, a period of one year. Regarding the decision-making process,the deregulation of European electricity markets makes that we concentrate on simulating the participation of the operatorwind generation is expected to be traded through electricity of the considered wind farm in the Dutch electricity market,pools, in which energy bids are settled several hours before composed by the day-ahead pool APX (standing for Amster-actual delivery. Imbalances lead to financial penalties that may dam Power eXchange) and by the real-time regulation marketsubstantially tighten the revenue of wind power producers. run by TenneT, the Transmission System Operator (TSO) forOwing to these two reasons, predictions of wind generation up the Netherlands. We compare several bidding strategies, basedto 48 hours ahead contribute to a secure and economic power on point predictions only, or alternatively on more advancedsystem operation. Increasing the value of wind generation strategies derived from probabilistic forecasts.through the improvement of prediction systems' performanceis considered as one of the priorities in wind energy research

II. PROBABILISTIC FORECASTS OF WIND GENERATIONneeds for the coming years [2].So far, most wind power prediction tools give an estimate AND THEIR REQUIRED PROPERTIES

of the future power generation without addressing the issue A. Definitionsof the forecast uncertainty [3]: we usually say that such In this paper, we consider that wind generation for a givenprediction systems provide the 'most likely outcome' for time t can be seen as a random variable Pt. Hence, theeach look-ahead time. Given the significant variability of measured power output p* is a realization of Pt. Denote bythe level of forecasting errors, end-users have expressed the ft the probability density function of that random variable.need for an on-line estimation of forecast uncertainty. This If a point forecasting method is developed for providingadditional information would then be introduced in decision- point predictions that would minimize a mean square errormaking processes. Such an information can indeed be given criterion, the resulting point forecast mt±an/tmade at timeby prediction risk indices that tell what may be the expected t for lead time t + k corresponds to an estimate of thelevel of forecasting error [4], or alternatively by quantile or expectation of Pt±kC Such an information could be seen as

P. Pinson is with the Informatics and Mathematical Modeling department of sufficient for the management or trading of wind generation.the Technical University of Denmark, Richard Petersens Plads, DK-2800 Kgs. But, associated uncertainty estimates would permit to optimizeLyngby, Denmark (e-mail: [email protected]). J. Juban and G. Kariniotakis are decisions resulting from the use of these predictions. Ideally,with the Center for Energy and Processes of Ecole des Mines de Paris, B.P. . of p wol giv thhlnomto207, F-06904 Sophia-Antipolis, France. (e-mail: [email protected]; an estimate Jt±k/t ofJt±k ol [email protected]). on what could be the wind generation at time t + k. Today,

©C Copyright KTH 2006Authorized licensed use limited to: Danmarks Tekniske Informationscenter. Downloaded on November 20, 2009 at 10:32 from IEEE Xplore. Restrictions apply.


probabilistic forecasts ofwind generation are given in the form coverage should exactly equal the pre-assigned probability.of quantile or interval forecasts, produced with statistical post- That first property is referred to as 'reliability' or 'calibration'processing methods [4], [6], or from wind power ensembles in the forecasting literature.[5]. Fig. 1 depicts an episode with hourly point predictions Besides this first requirement, it is necessary that predic-for the following 48 hours, given by a state-of-the-art fore- tion intervals provide a situation-dependent assessment of thecasting approach, compared to the measured power values. forecast uncertainty. Their size should then vary dependingThe prediction intervals, which are estimated consequently by on various external conditions. For the example of windthe adapted resampling method, are shown here in the form prediction, it is intuitively expected that prediction intervalsof a fan chart. Hereafter, all variables are normalized by the (for a given nominal coverage rate) should not have the samenominal power Pn of the considered wind farm. size when predicted wind speed equals zero and when it is near

cut-off speed. The most simple type of intervals is constant-100 size intervals (e.g. produced from climatology). Advanced

90% int. methods for their estimation are expected to produce variable-80 80% int. size intervals. This property is commonly named 'shar-pness'

'Em | |670%int. or 'resolution'. Note that here, we will introduce a nuance

n M~~~~~~~~~~60%int.he ,0 60 L 50% int. between sharpness and resolution: the forner will relate tool 40% it. the average size of intervals while the latter is associated to

in30% int.40 m2.............their size variability.

Ml 0% int. Actually, the traditional view of interval forecast evalua-20-pred. tion, which mainly comes from the econometric forecasting

community, is based on the testing of correct conditional0 5coverage. This means intervals have to be unconditionally

look-ahead time [hours] reliable, and independent (see [9] for instance). In the caseof wind power forecasting, we know there exists a correlation

Fig. 1. Example of a 48-hour point prediction of wind generation, associated among forecasting errors (at least for short time-lags). Thus,with a set of interval forecasts. The point predictions are given by a state-of-the-art method and interval forecasts are estimated consequently with the we do not expect prediction itervals to be idependent. Then,adapted resampling approach. it appears preferable to develop an evaluation framework that

is based on an alternative paradigm. We propose to considerPrediction intervals give the range of possible values within reliability as a primary requirement and then sharpness and

which the true effect is expected to lie with a pre-assigned resolution as an added value. It should be noted here that reli-probability, known as their nominal coverage rate. Here we ability can be increased by using some re-calibration methodsconcentrate on central prediction intervals t+) estimated (e.g. conditional parametric models [5] or smoothed bootstrapat time t for lead time t + k with a nominal coverage rate [10]), while sharpness/resolution cannot be enhanced with(1 - a), the bounds of which correspond to the (o/2) and post-processing procedures. This second aspect is the inherent(1 - a/2) quantiles of the predictive distribution fI±k/t of (and invariant) ability of a probabilistic forecasting method toexpected power generation at that lead time: distinctly resolve future events [11].

-[t+klt - t+k1rt+k(t *) II. EVALUATING THE QUALITY OF WIND POWEREstimating at once a sequence of n interval forecasts with PROBABILISTIC FORECASTS

various nominal coverage rates varying over all values in the A Focus on reliabilityunit interval permits to construct predictive distributions ofwind generation. Denote by a = {ai,aa2,.., a,} the set of Firstly, let us introduce the indicator variable k whichnominal coverage rates (with the ai in the ascending order). is defined for a quantile forecast t±k / made at time t andProbabilistic wind power forecasts made at time t for lead time for horizon k as follows:t + k are then given by the set of corresponding 2n predictive 1 i PL± .quantiles: T otherwise t+k/lt (4)

P - ,-1/2)(1-cj/2) <1-%) i /2)ftl+klt = Lrt+kl/t x rt+kc/t : t+k/t:(1-aj/2) r (1 , 1/2) r(1-aj/2)'} (2) Consequently, denote by nm) the sum of hits and nh) thet+kl/t : * * *: t+kl/t : t+kl/t : sum of misses (for a given horizon k) over the N realizations:

which can also be written as N

fL±k/t -{r+kG t+/t'j *,&tk (3) Thk.1 = ,k=1} = E k (5)

(° ) -_ f (a)0l--) ( 6)B. Required properties Thk, #{Itkt O} NJ -in (6)

The first requirement for interval forecasts is that their mea- The easiest way to check the calibration of probabilisticsured coverage should be close to the nominal one. Actually, if forecasts is to compare the empirical coverages of the variousconsidering infinite series of interval forecasts, that empirical quantiles to the nominal ones (i.e. the required probability



di). An estimation a(a of the actual coverage a(a), for a should reward a forecaster that expresses his true beliefs. It isgiven horizon k, is obtained by calculating the mean of the said to be proper if it does so. Gneiting et al. [16] recently{t)}ki.N time-series over the test set: showed that any scoring rule of the form

i~~~) 1 m5j~~~~~~ (7) 2m + (sdPOei)ak HN ~ <)*nk (7) S( tp+k/tP±k) S (iSi( /t) + (Si(Pt+k)Ntln d) + n d) =t=1 ~ ~k,O +Tk,1 i

, _ Kr(i) aa(i) ~~~~~~+hPt*k) O{1)This standard approach to the evaluation of prediction si( t+k/l)) t, + ) (intervals was proposed by Ballie et al. [12] and by McNees with the indicator variable (for the quantile with pro-[13]. This is the idea used in reliability diagrams which t,kgive the empirical coverage versus the nominal coverage for portion di) introduced above, si non-decreasing functions and

h arbitrary, was proper for evaluating predictive distributionsvarious nominal coverage rates. The closer to the diagonal e a s a set foq uanties Thefincoe valueisthe better. They can alternatively be depicted as the deviation nexpressed as a set of quanvles. The final score valueiSfrom the 'perfect reliability' case for which empirical coverage dbtins overathe e set, asua fon of thedlook-would equal the nominal one (calculated as the difference .

the'

between these two quantities). This idea is similar to the use of ahead time k. Here S is a positively-oriented score: a higherProbability Integral Transform (PIT) histograms as proposed score value stands for a higher skill.

...14 exceptthatreliabilit diagrams diA unique score does not tell what are the contributionsin[14], except that reliability diagrams directly provide that of reliability or sharpness/resolution to the skill of agieadditional information about the magnitude of the deviation probabilit forecastngsmesthod becaue sitlnompassesfrom~~~~~th.pretrlaiiy ae

probabilistic forecasting method, because it encompasses allthe aspects of probabilistic forecast evaluation (see discussionin [15]). Though, if reliability is assessed in a first stage, then

B. Focus on sharpnesslresolution relying on skill scores such as the one given by Eq. (11) allowsWhen dealing with sharpness or resolution, focus is given one to compare the overall skill of rival methods.

to the size of the prediction intervals. Let us define

6(a) _ (1-a/2) A(a/2) C Evaluation of the quality of the probabilistic forecastst,k t+k/t rt+k/t (8) obtained by adapted resampling

the size ofthe central interval forecast It(+)/. Adapted resampling is a statistical distribution-free ap-If two uncertainty estimation methods provide intervals at proach to the estimation of quantiles and interval forecasts

an acceptable level of reliability, it is the method that yields suitable for non-linear and bounded processes [4]. It servesthe narrowest intervals that is to be preferred. Hence, we relate to post-process point forecasting methods and enhances themthe sharpness aspect to the average size (a) of the prediction with probabilistic forecasts of wind generation. This approachintervals for a given horizon k: has been applied and evaluated on a variety of case-studies

N N consisting of periods ranging from several months to sev-

k(a) _ 1 E 6(a) 1 E ((1 a/2) _(a/2) (9) eral years for various European wind farms. Here, adaptedHk Zt,-k H Zt Kt+k/t t+k/l) ( resampling is applied for post-processing point predictions

provided by 3 methods from different research centers inIn parallel, the resolution concept is standing for the ability Europe (denoted by MI, M2, and M3). The evaluation period

of providing a situation-dependent assessment of the uncer- is composed by 8760 series of 43-hour ahead hourly predic-tainty. It is not possible to directly verify that property, though tions. Predictive distributions are built by estimating 9 centralwe may study the variation in size of the intervals. The predictions intervals with nominal coverage rates 10, 20,..., andstandard deviation a (a) of the interval size (for a given horizon 900. Fig. 1 gives an example of such series of probabilistick and nominal coverage rate (1- a)) calculated as forecasts of wind generation. Regarding the settings of the

(N - 2) method, we follow the sensitivity analysis carried out in [15]:(cr) _ P 1 E (a) ((,10) we set the sample size to 300 items, the number of bootstrapk tN I tk-1iV (I, ) replications to 50, and we use 5 triangular fuzzy sets for

mapping the range of power values.provides that information. Because of the non-linear and For asessingin a ofirsagethconditionally heteroskedastic nature of the wind generation Fofpoabistic peictios,wtae use reliability diagram whichprocess, the forecast uncertainty is highly variable and it is th deviatio from 'erfe reliability asramfunctthus expected that the interval size also greatly varies. The thenom iaioverag raesftelqanilig. 2) Deviation_(O) (O) . . ~~~~~~~thenominal coverage rates of the quantiles (Fig. 2). Deviationway both dk and (Tk can be considered for evaluating the values are the averages over the 43 look-ahead times. Thequality of predictive distributions is described in [15]. figures in the legend give the average absolute deviations

Alternatively, one may envisage to use skill scores for eval- ovrteanefnmilcvrgeaesOesesrmuating the sharpness and resolution of probabilistic forecasts. Fig. 2 that the deviations from nominal coverage are comprisedSkill scores are obtained from scoring rules that associate a between -1.8 and 0.30% whatever the point forecasts used assingle numerical value S( ftP±k/t' Pt±k) to a predictive distri- input. Owing to these low deviations from 'perfect reliability',bution fjP±kt if the event Pt+±k materializes. A scoring rule we assume that the three sets of probabilistic forecasts are



well-calibrated. However, there is a difference in the overall -1reliability of predictive distributions: the ones produced fromthe Ml forecasts are more reliable than the two others. _15 E Ml (-2.6301)Predictive quantiles resulting from M2 and M3 forecasts are M2 (-2.6939)slightly underestimated.

=3 -2(ua)

0.5 ~~~~~~~~~~~~~0

35-

0 r 0 5 10 15 20 25 30 35 40 45horizons [hours]

U> -

Fig. 3. Skill score values as a function of the prediction horizon for the three

-1.5 ideal sets of probabilistic forecasts. Figures in the legend give the average scoreA- Ml (0.30842) values over the range of look-ahead times.

M2 (0.6861)

-21 M3 (0.46441)_0 20 40 60 80 100

nominal coverage rate [%] Here, we consider that all energy producers participate inthe electricity market under the same rules, i.e. they have to

Fig. 2. Reliability diagrams for evaluating the reliability of the three series propose their bids on the day-ahead market and they are thenof probabilistic forecasts of wind generation. Figures in the legend give the propose teisons the day-heade matrkt adheyae. Thenaverage absolute deviation from 'perfect reliability' over the range of nominal financially responsible of their deviations from schedule. Thecoverage rates. costs of keeping the balance are charged to the participants,

proportionally to their imbalance. A description of APX andIn a second stage, because we have assumed the three TenneT markets is given in [17]. In the Netherlands, bids are to

sets of probabilistic forecasts to be well-calibrated, we turn be given before 10:30 for the following day from midnight toour attention to the use of the skill score introduced above midnight. Relevant predictions horizons are then between 14(Eq. (1 1)), for assessing their overall skill. We put s i(p) = 4p, and 38 hours ahead (if the last available forecast is provided at(i 1,... 2n), and h(p) = -2p, following [16]. Fig. 3 10:00). The length of the Program Time Unit (PTU) on APXdepicts the evolution of the resulting score, as a function of is of 1 hour. But then, the PTU length on the regulation marketthe look ahead time. Figures in the legend give the average run by Tennet is of 15 minutes only. Therefore, we considerscore values over the range of horizons. There is a trend that that bids proposed on APX are composed by four equalthe score values decrease for longer horizons. This meets the amounts of energy for each of the TenneT PTUs composinggeneral statement that it is harder to forecast events that are an APX trading hour. Wind energy predictions are obtainedfurther in the future. One notices that the skill of the three by integrating the power forecasts over the related PTUs.sets of probabilistic forecasts is rather close. But, even if the A crucial assumption for this study is that the wind poweraverage score values tell that MI predictive distributions have producer is a price-taker in the Dutch electricity market. Thishigher skill than the two others on average, one sees that the means that the amounts of energy he proposes on the marketbest and worst predictive distributions are not obviously the cannot impact the market clearing price 7w. We formulate thesame over the whole range of horizons. Also, although M3 same assumption regarding the regulation market: the windprobabilistic forecasts are more reliable than the M2 ones, the farm operator is a small entity on that market and have nooverall skill of the latter is slightly higher. This is because influence on imbalance prices.predictive distributions obtained from M2 point predictions Whatever the considered electricity market, the revenue R kprove to be sharper. for a given PTU k of a participant proposing an amount of

energy E' but actually generating E7± can be formulated asIV. EVALUATING THE VALUE OF WIND POWER

PROBABILISTIC FORECASTS Rk(E,E7±k) - Tk (E',E7±k) (12)For studying the operational value of predictive distributions This revenue is composed by the income resulting from the

of wind generation, we tur our attention to the simulation of selling of actual wind generation at the spot price, minus thethe participation of the operator of the considered wind farm cost TZ (E2, E±k) associatedtin the Dutch electricity market over the year 2002. d d as

A. Assumptions and problem formulation dk =E7±k - Ek (13)Each electricity pool has its own rules and regulations In turn, the regulation costs are a function of the spot price

that determine the way energy is to be sold or purchased, qrk, the regulation unit costs <r) and wr)' for downwardhow the market prices are settled, and the obligations that and upward dispatch respectively, and the amount of energythe participants (producers or consumers) are committed to. produced in imbalance dkc:



higher than the former over the first six months. One noticesT*( c2kI*d_ f k ,dk >. 14) that the unit cost for positive deviations from contract is on

V kkk , dk < average higher than the unit cost for negative deviations. Theaverage values for 7`) and 7*r+ over 2002 are of 4.03 and

B. Definition of bidding strategies 10.93E/MWh respectively. Hence, it would appear preferableDifferent types of bidding strategies may be defined depend- to overestimate expected wind generation.

ing on the available information about future wind generation. TABLE IIf only point predictions are provided by a forecasting method, DUTCH ELECTRICITY MARKET TABLERIDUTCELCTRCIT MAKETCHARACTERISTICS FOR 2002: QUARTERLYthen these point predictions appear as the best bids one may AVERAGES OF SPOT PRICES AND REGULATION PENALTIES FOR UPWARDpropose on the electricity pool [18]. However, if the market

AND DOWNWARD DISPATCH.participant has predictive distributions, he can develop moreadvanced strategies, reflecting either his aim to maximize his Quarter 7rAc [e/MWh] wIT ' [E/MWh] wIF*'+ [e/MWh]revenue over a certain period of time or a risk aversion for 1 11.65 0.33 16.22large regulation costs [19], [20]. Here we concentrate on the 2 38.38 1.34 11.13first type of strategies, referred to as Probabilistic Choice (PC) 341.17 8.22 8.51strategies. 4 29.38 6.97 7.61

Applying a PC strategy directly translates to maximizingthe expectation of the revenue for each PTU k, formulated byEq. (12). Because in that formulation all wind generation is For setting up the model of regulation penalties, we assumesold at the spot price, and the level of contracted energy only it is possible to perfectly predict these quarterly averages. Thisinfluences the regulation costs, an optimal energy bid Ec is consequentlyyields four differentmodels ofthe regulationunitdetermined by minimizing the expectation of these costs: costs, depending on the quarter of the year. Besides that, wedetermined byminimizing the expetationofthesecosts: consider the three sets of probabilistic forecasts, the quality

E=argmin E [TZ(E, Et+k)], (15) of which have been assessed in the previous Section. Out ofECk the 8760 predictions series used for the quality evaluation,

where this expectation can be written as only 365 are actually considered for defining optimal bidsEkc on the electricity pool. They are the forecast series produced

E [TZ(Ek, EL±k)1 -X)fi+k(x)dx everyday at 10:00. The PC strategies resulting from the use of

1 predictive distributions obtained by post-processing MI, M2,+ J < +(x - Ec)fk(x)dx. (16) and M3 are denoted by PCI, PC2, and PC3 respectively. For

EC k k fi+k (x) dx. (16)comparison, participation strategies directly derived from the

Because neither the regulation unit costs nor the probability point predictions are also evaluated over 2002, and denoted

distributions of future wind generation can be known when by NI, N2 and N3.proposing bids on the electricity pool, it is necessary to Results from the application ofthe various bidding strategies

consider.probabilisticforecass as well as a model of are gathered in Table II. In this Table,r1corresponds to thethe pa.rtcpan sensitivt toregulati+kot T l overall quantity of energy that is traded on the regulationbe d erivedfrom estimate o regulation utstis ly,tor market, in percentage of the total amount of energy producedmore genivedfrall modetinathes wegulaythe wnin powe ,proder over that year. It is then decomposed in the parts that aremore generally by modeling the way the wind power producer sujc todwwr +

an upadq ipth(uhtawill face deviations from contract in the most cost-efficient j t w a upward ratispatch (such thamanner, if he has the possibility to couple his production with +r + rE. r r

. 1~~~ncome of the wind farm operator as a percentage of theconventional generation, or to use storage devices. Here, weconsider regulation penalties only in this model. The way the revenue he would have obtained by using perfect forecastsoptimization problem given by Eq. (15) integrates these two (and thus without any cost for regulation).information, and is consequently solved, is described in [20]. TABLE II

RESULTS FOR THE VARIOUS BIDDING STRATEGIES.

C. Results and discussionTable I gathers the quarterly averages of both the market NI N2 N3 PCI PC2 PC3

clearing prices and regulation unit costs for upward and 23.96 23.30 23.33 15.73 15.97 17.17downward dispatch over 2002. The average spot price on APX - [0] 23.45 23.50 26.36 45.75 45.99 43.96proves to be significantly variable throughout the year, being %[] 47.41 46.80 49.69 61.48 61.96 61.174 times higher during the third quarter (corresponding to the ~ [0/] 84.88 83 574 0.44 9. 90.08summer period) than during the first one (corresponding inturn to the winter period). The average market clearing price One sees from Table II that whatever the considered pointover 2002 is equal to 29.99e/MWh. In parallel, the regulation prediction method, applying the more advanced strategy in-market behavior exhibits also substantial variations. Although stead of using the point forecasts only permits to augment thethe average unit costs for upward and downward dispatch are market participant's income. The revenue of the operator israther similar for the last two quarters, the latter is much increased by 5-6 points, which corresponds to reducing the



regulation costs by one third over that year. It is not achieved REFERENCESby decreasing the amount of energy subject to the regulation [1] A. Zervos, "Developing wind energy to meet the Kyoto targets inmechanism this quantity is substantially higher but by the European Union," Wind Energy, vol. 6, no. 3, pp. 309-319, July-orientating the imbalances. While r1+ and rj- are at similar September 2003.

[2] S.-E. Thor and P. Weis-Taylor, "Long-term research and developmentlevels for the bidding strategies Ni, N2 and N3, the latter needs for wind energy for the time frame 2000-2020," Wind Energy,is approximately three times higher than the former when vol. 5, pp. 73-75, April-June 2003.applying the more advanced bidding strategies. [3] G. Giebel, R. Brownsword, and G. Kariniotakis, "State of the art

Predictive distributions produced .tom MI were found to be on short-term wind power prediction," June 2003, Anemos Project,Predictive distributions produced from M1 were found to be Deliverable Report DI. 1, available online: http://anemos.cma.fr.the most reliable and to have the highest overall skill. Here, [4] G. Kariniotakis and P. Pinson, "Uncertainty of short-term wind powerthis translates to a higher value from their use when bidding forecasts - A methodology for on-line assessment," in Proc. of the

IEEE/PMAPS Conference 2004, 8th Conference on 'Probabilistic Meth-on the Dutch electricity market. For the two other series of ods Applied to Power Systems', Ames, Iowa (USA), September 2004,probabilistic forecasts, the ones (M3) that were more reliable pp. 729-736.had a lower skill. The performance ratios corresponding to [5] H. A. Nielsen, T. S. Nielsen, H. Madsen, and K. Sattler, "Windtheir related bidding strategies are almost equal. In general, the power ensemble forecasting,," in Proc. of the 2004 Global WindpoiwerConference, Chicago, Illinois (USA), March 2004.differences between a values for the three advanced bidding [6] H. A. Nielsen, T. S. Nielsen, and H. Madsen, "Using quantile regressionstrategies are lower than the ones between strategies NI, N2, to extend an existing wind power forecasting system with probabilisticand N3. forecasts," Wind Energy, vol. 9, no. 1-2, pp. 95-108, January-April 2006.[7] P. Pinson and G. Kariniotakis, "On-line assessment of prediction risk

for wind power production forecasts," Wind Energy, vol. 7, no. 2, pp.V. CONCLUSIONS 119-132, May-June 2004.

Several methods exsoa o h r[8] A. H. Murphy, "What is a good forecast? An essay on the nature ofSeveral methods exist today for the probabilistic forecasting goodness in weather forecasting," Wea. Forecasting, vol. 8, no. 2, pp.

of wind generation. Such a kind of forecasts provide the whole 281-293, 1993.information on what could be the wind power production [9] P. F. Christoffersen, "Evaluating interval forecasts," International Eco-

nomic Review, vol. 39, no. 4, pp. 841-862, 1998.in the following hours (typically up to 48-hour ahead). We [10] P. Hall and A. Rieck, "Improving coverage accuracy of nonparametrichave investigated on two aspects of the evaluation of pre- prediction intervals," J Royal Stat. Soc., vol. 63, no. 4, pp. 717-725,dictive distributions: their quality, which corresponds to their 2001.statistical performance, and their value, which relates to the [11] Z. Toth, 0. Tallagrand, G. Candille, and Y Zhu, "Probability and

ensemble forecasts," in Forecast Verification: A Practitioner's Guide inresulting benefits from their use in an operational context. This Atmospheric Science, I. Jolliffe and D. Stephenson, Eds. John Wileyevaluation was based on three series of probabilistic forecasts & Sons Ltd, 2003.

[12] R. T. Baillie and T. Bollerslev, "Prediction in dynamic models with time-obtained by the adapted resampling approach. dependant conditional variances," J Econometrics, vol. 51, pp. 91-113,

The quality of predictive distributions of wind generation 1992.has been assessed in term of their reliability and sharp- [13] S. McNees, "Forecast uncertainty: can it be measured?" Federal Reserve

Bank of Boston, Boston" mimeo, 1995.ness/resolution. Here, we have shown that the three forecast se- [14] T. Gneiting, F. Balabdaoui, and A. E. Raftery, "Probabilistic forecasts,ries could be considered as reliable, and that their overall skill calibration and sharpness," University of Washington, Department of(then encompassing both reliability and sharpness/resolution) Statistics, Tech. Rep., May 2005, technical report no. 483.

[15] P. Pinson, H. A. Nielsen, T. S. Nielsen, G. Kariniotakis, and H. Madsen,was pretty similar. ;"Properties of quantile and interval forecasts of wind generation and

Then, the evaluation of their value showed firstly that their evaluation,," in Proc. of the 2006 European Wind Energy Associ-considering bidding strategies derived from probabilistic fore- ation Conference, EWEC'06, Athens, Greece, February 2006, Scientific

casts alows oeto inrease he reveue reslting fom his Track.casts allows one to increase the revenue resulting from his [16] T. Gneiting and A. E. Raftery, "Strictly proper scoring rules, prediction,participation in an electricity pool. Moreover, the predictive and estimation,:" University of Washington, Department of Statistics,distributions that proved to be the most reliable and skillful Tech. Rep., 2004, technical report no. 463.

[17] A. Boogert and D. Y Dupont,, "On the effectiveness of the anti-gaminghad a higher value for trading through the electricity market. policy between the day-ahead and real-time electricity markets in theNote that the value of probabilistic forecasts does not lie Netherlands,," Energy Economics, vol. 27, no. 5, pp. 752-770, September

only in this revenue-maximization aspect but also in the 2005.fact that the resulting trading or management strategies of [18] J. Usaola, 0. Ravelo, G. Gonzalez, F. Soto, C. Davila, and B. Diaz-

Guerra, "Benefits for wind energy in electricity markets from usingwind generation can be specially tailored to the forecast user short-term wind power prediction tools - A simulation study," Windneeds. This is hardly possible if relying on point predictions Engineering, vol. 28, no. 1, pp. 119-128, 2004.

[19] G. N. Bathurst, J. Weatherhill, and G. Strbac, "Trading wind generationonly. In the future, we will further develop on methodologies in short-term energy markets," IEEE Trans. on Power Syst., vol. 17,based on predictive distributions, in a stochastic programming no. 3, pp. 782-789, August 2002.framework, for an optimal management of wind power. [20] P. Pinson, C. Chevallier, and G. Kariniotakis, "Optimizing benefits from

wind power participation in electricity markets using advanced toolsfor wind power forecasting and uncertainty assessment," in Proc. of

ACKNOWLEDGMENT the 2004 European Wind Energy Association Conference, EWEC'04,This work was performed in the frame of the ANEMOS London, United Kingdom, November 2004.

Project (ENK5-CT2002-00665) funded in part by the Euro-pean Commission and by ADEME, the French Environmentand Energy Management Agency. The authors gratefully ac-knowledge ELSAM for providing data for the realization ofthe study. Acknowledgements are finally due to the modelerswho provided their wind power point predictions.



Pierre Pinson was born in Poitiers, France. He ences (INSA Rouen) in 2004. He spent his last academic year at McGillreceived the M.Sc. degree in applied mathematics University (Canada). He is currently a Ph.D. student at the Center for Energyfrom the National Institute for Applied Sciences and Processes of Ecole des Mines de Paris. His research interests include(INSA Toulouse) in 2002 and the Ph.D. degree in artificial intelligence, forecasting methods, risk assessment, and renewableenergetics from Ecole des Mines de Paris in 2006. energies.Part of his education was made at University ofLeicester (UK), TU Delft (The Netherlands), and theTechnical University of Denmark. He is currentlywith the Informatics and Mathematical Modelingdepartment of the Technical University of Denmark Georges N. Kariniotakis was born in Athens, Greece. He received hisas a junior scientist. His research interests include Engineering and M.Sc. degrees from the Technical University of Crete, Greece

among others forecasting methods, optimization under uncertainty, decision and his Ph.D. degree from Ecole des Mines de Paris in 1996. He is currentlysciences, and renewable energies. with the Center for Energy and Processes of Ecole des Mines de Paris as a

senior scientist. He is the scientific and technical coordinator of the EU projectAnemos. His research interests include among others renewable energies, wind

Jeremie Juban was born in Saint-Etienne, France. He received the M.Sc. power forecasting, distributed generation and artificial intelligence. He is adegree in applied mathematics from the National Institute for Applied Sci- member of IEEE.


Documents

On the quality and value of probabilistic forecasts of ... · 9th International Conference on Probabilistic MethodsApplied to PowerSystems KTH, Stockholm, Sweden-June 11-15, 2006