CMEArrivalTime&ImpactWorkingTeam

ChristineVerbeke,LeilaMays,SandroTaktakishvilli

CMEArrivalTime&ImpactWorkingTeam

✳ EvaluatewherewestandwithCMEarrivaltimeandimpact prediction✳ Establishcommunity-agreedmetricsandeventsregardingCMEarrivaltimeandimpact.✳ Provideabenchmark againstwhichfuturemodelscanbeassessedagainst✳ Complementarytothe CMEScoreboard (collectanddisplayreal-timeCMEpredictionsandfacilitatethevalidationofreal-timepredictions).

✳ Catalogofmetrics andhowtheyrelatetouserneedsandscienceneeds.✳ Modelassessments withselectedmetricsforselectedtimeintervals.✳ Onlinedatabaseofmodelinputs,outputs,andobservations.✳ Publication describingmodelassessmentresultssummarizingwherewestandwithCMEarrivaltimeandimpactprediction.

Work with Information Architecture for Interactive Archives (IAIA) working teamfor online database.

TeamGoals

TeamDeliverables

CMEArrivalTime&ImpactWorkingTeam

✳ Identifyanddiscussuserneeds✳ Discussandselecttimeintervalstostudy— expandasneeded✳ Discussanddevelopasetofrelevantskillscores,andrelatethemtouserneeds andscienceneeds✳ Identifysourcesofuncertainty✳ Producemodel/techniqueoutputforintervalsofstudy✳ Performmodelassessments withselectedmetrics

✳ Slackchannels(contactleadstobeadded)✳ Mailinglist(contactleadstobeadded)✳ Email✳ Telecons✳ Regularwebsiteupdateshttps://ccmc.gsfc.nasa.gov/assessment/topics/helio-cme-arrival.php

Summaryofteamtasks

Remotecollaboration

CMEArrivalTime&ImpactWorkingTeam

Participants– inviteyourcolleagueshighinterestEricAdamson*·TanjaAmerstorfer ·Anastasios Anastasiadis ·NickArge ·MichaelBalikhin*·DavidBarnes*·Francois-XavierBocquet ·Yaireska Collado-Vega*·PedroCorona-Romero*·JackieDavies·CurtdeKoning*·CraigDeForest*·Manolis K.Georgoulis*·CarlHenney ·BernardJackson*·LanJian·MashaKuznetsova*·Kangjin Lee·Noé Lugaz ·AnthonyMannucci*·Periasamy KManoharan*·DanielMatthiä*·LeilaMays*·MikeMcAleenan*·Slava Merkin*·Marilena Mierla ·JosephMinow*·ChristianMoestl ·KarinMuglach*·TeresaNieves·Nariaki Nitta·MarlonNunez·Dusan Odstrcil*·MathewOwens·Evangelos Paouris ·AthanasiosPapaioannou ·SpirosPatsourakos ·vic pizzo ·PeteRiley·AlexisRouillard·CamillaScolini ·HowardSinger*·RobertSteenburgh*·Aleksandre Taktakishvili*·ManuelaTemmer·W.KentTobiska*·ChristineVerbeke*·Angelos Vourlidas ·KatherineWinters*·AlexandraWold*·KiChang Yoon·Emiliya Yordanova*·Jie Zhang·mediuminterestTarekAl-Ubaidi*·SuzyBingham*·StevenBrown*·BaptisteCecconi ·DavidFalconer·NataliaGanushkina*·LauraGodoy*·BerndHeber·ChristinaKay·AdamKellerman*·BurcuKosar*·AlexanderKosovichev*·YukiKubo·PeterMacNeice*·Chigomezyo Ngwira*·StevePetrinec*·NikolaiPogorelov*·LutzRastaetter*·IanRichardson*·NeelSavani*·BarbaraThompson*·Karlheinz Trattner*·RodneyViereck ·BrianWalsh·Chunming Wang*·DanielWelling*·Yongliang Zhang*·Yihua Zheng*·

*attendingCCMC-LWSworkingmeeting

CMEArrivalTime&ImpactWorkingTeam

Participatingmodels✳ DBM(Vrsnak &Zic)✳ ElEvo (EllipseEvolution)(Moestl)✳ ElEvoHI (EllipseEvolutionbasedonHI) (Amerstorfer)✳ Enhanceddrag-basedmodel(Hess&Zhang) [set1results]✳ EUHFORIA (Pomoell)✳ SARM(Núñez) [set1results]✳ SUSANOO-CME✳ WSA-ENLIL+Cone (Arge,Odstrcil) [set1results]✳ contactustoaddyourmodel

APRIL4•TUESDAY9:00am– 10:15amMetrics•Metricsdiscussionandexamples•Metricsaddressinguserneedsvs.scientificresearch•Discussionquestions•1stsetofeventsforvalidation•Preliminarymodelvalidation/results•AlexWold:Real-timeENLILrunvalidation&discussion•Marilena Mierla (notattending):comparisonsofENLILandEUHFORIA•More2-3slidecontributionsfromparticipants•CMEscoreboarddiscussion•SuzyBingham:InitialCMEscoreboardverificationfromtheUKMetOffice

TeamAgendaMetrics&Impactofbackgroundsolarwind•ContinueitemsremainingfromMetricssession•Scenesettingpresentationonquantifyingtheeffectsofbackgroundsolarwind•Discussiononimpactofbackgroundsolarwind,2-3slidecontributionsfromparticipants•Continueotherdiscussionitemsfromprevioussessions

Summaryandfutureplans•Remainingdiscussionquestions•QuantifyingprogressinthefieldofCMEarrival&impact•Summaryofteamprogress•Futureplans,meetings,remotecollaboration

APRIL5•WEDNESDAY10:45am– 12:00pm

APRIL6•THURSDAY4:45pm– 6:00pm

APRIL4•TUESDAY9:00am– 10:15amMetrics•Metricsdiscussionandexamples•Metricsaddressinguserneedsvs.scientificresearch•Discussionquestions•1stsetofeventsforvalidation•Preliminarymodelvalidation/results•AlexWold:Real-timeENLILrunvalidation&discussion•Marilena Mierla (notattending):comparisonsofENLILandEUHFORIA•More2-3slidecontributionsfromparticipants•CMEscoreboarddiscussion•SuzyBingham:InitialCMEscoreboardverificationfromtheUKMetOffice

TeamAgendaMetrics&Impactofbackgroundsolarwind•ContinueitemsremainingfromMetricssession•Scenesettingpresentationonquantifyingtheeffectsofbackgroundsolarwind•Discussiononimpactofbackgroundsolarwind,2-3slidecontributionsfromparticipants•Continueotherdiscussionitemsfromprevioussessions

Summaryandfutureplans•Remainingdiscussionquestions•QuantifyingprogressinthefieldofCMEarrival&impact•Summaryofteamprogress•Futureplans,meetings,remotecollaboration

APRIL5•WEDNESDAY10:45am– 12:00pm

APRIL6•THURSDAY4:45pm– 6:00pm

CMEArrivalTime&ImpactWorkingTeam

ConsiderationsforEventSelectionConsiderationsforeventselection:✳ SingleCMEevents(fastandslow)✳ MultipleCMEevents(interactingandnon-interacting)✳ CMEeventsthatareexpectedtoarrivebutdonot(falsealarm)✳ FlankimpactCMEevents✳ ConsidereventsfromtheISESTWG4wikipage✳ Overlapsomeeventswiththe IMFBzandL1workingteam

✳ Forsomevalidationmethods,howmanyeventsareneededtobestatisticallysignificant?✳ Eventselection:Shouldwehavea"trainingset","validationset",and"testset"— wherethe"test"setisnotrevealeduntilalaterstage?

CMEArrivalTime&ImpactWorkingTeam

1st setofeventssmallcoreselectiontoexplorechosenmetrics&validationtechniques

• Fourevents:twohits,oneproblematichit,andonefalsealarm.

• Aimfor2hitstowilloverlap withtheIMFBz workingteam’seventlisttoreducetheoverallmodelingburden(forthosemodelsthatpredictbotharrivalandBz).

• Ifdesired,theCMEparametersprovidedonthewebsite(takenfromliterature)canbeused

A)3April201010:33UT(hit)B)15March201307:12UT(hit)C)7January201418:24UT(falsealarm;onlyaweakdiscontinuityarrives)D)15March201501:48UT(hit;problematic,manymodelspredictalatearrival)

Resultsfrom4modelsfortheseeventshavebeenpostedonourwebsite:https://ccmc.gsfc.nasa.gov/assessment/topics/CME/events.php#results

• ThevalidationofCMEforecastingfocusedaroundarrivaltimebecauseitwastheeasiestthingtoagreeon.Longtermgoalistovalidateintensity andduration also.

• Importanttorealizethatquantitiestovalidateforresearchisdifferentoperations.E.g.littleinterestinT or nverificationforoperations.

• Scienceresearchfocusisonhowwellmodelforperformsformostparameters

• Foroperations themostimportantquantitiesareBz andv (couplingfunction)

• Quantitiesimportantinthisorderforoperations:timing(arrival),intensity,andduration

Validation:UserNeedsvsResearchNeedsfeedbackfromSWPC

Validation:UserNeedsfeedbackfromSWPC

Quantitiestovalidate:• timing(arrival),intensity,andduration

• Arrivaltime:powergridisinterestedintiming

• SuggestusinghistoricICMEobservationstodeterminewhattovalidate(e.g.whatintensity,duration,orthresholdisimportanttotestmodelperformance)

• Kp usedbyforecasters,butitmightbeusefultocompareDstpredictionsformodelsfortheirperformanceforecastingofstormintensityandduration.

• Alwaysusefultoprovideameasureofuncertainty

• Abest/worstcasescenarioisusefulforuserstomakedecisions

✳ WhataretheeffectsofthemodelinputsontheCMEarrivaltimeandimpacts✳ modelparameters✳ CMEparameters✳ inputmagnetograms✳ …

✳ Isthereasolarcycledependenceonmodelperformance?

✳ DowewanttofixtheCMEinputparametersandinputmagnetograms(ifapplicable)forallmodels?

Ifso,istheteamcomfortablefortheCMEparameterstobedeterminedbyanexpertthatisnotamodelerintheCMEArrivalTimeWorkingteamtoremovebias?

✳ Whataresomegoodtechniquestodeterminetheuncertainty/confidence ofthearrivaltimeprediction?

Discussionquestions:models

QuantitiestoValidate:someideastostartICME...✳ arrivaltime✳ averagemagneticfieldmagnitude✳ averagetemperature✳ averagespeed✳ duration✳ resultinggeomagneticstormstrengh (Kp,Dst,...)

✳ Arrivaltime:☼ RMSE,meanabsoluteerror(MAE),meanerror(ME),others?

✳ Categorical(yes/no)predictions:☼ skillscoresbasedoncontingencytables☼ probabilisticandcontinuouspredictionscanbeconvertedtocategoricalusingthreshold

✳ Probabilisticpredictions:☼ Reliabilitydiagram,BrierSkillScore,...

Skillscores/metrics

Discussion

✳ Whichcatalog touseforICMEarrival?Takeanaverage?

✳ Overwhatintervalshouldaveragein-situobservationsbederived?Useacatalog?

✳ Directlycomparetimeseriesw/observationsforsomemodels?Time-shiftmodelresults?

✳ Alsovalidatethethemagneticcloudarrivalinadditiontotheshock/discontinuity?

✳ Validatean"impactparameter"extractedfrommodelresults?Comparetoin-situfluxropefitparameters?

✳ Howcanwevalidateandquantifytheeffectofthebackgroundsolarwindpredictiononthearrivaltimeprediction?

✳ HowdointeractingormultipleCMEevents,orSIR+CMEeventsimpactthechosenmetrics?Howtoquantifymodelperformancefortheseevents?

Discussionquestions:Quantities&Observationswhattoquantitiestovalidate,andwhattocomparethemto

✳ Whatisagoodbaselinemodelor climatologytocompareagainst?

✳ Forthehitcalculation:☼ Howtodefineacategoricalyes/nofor"modelpredictedarrival"- human

analysisofmodelresultsoralgorithm?Whatanalysismethod?

☼ IfthemodelpredictedarrivaltimeismorethanxnumberofhoursfromtheobservedCMEarrivaltimeisitahit?Orafalsealarmandmiss?Varythedefinitionofthehitdependingonuserneeds?

✳ Probabilisticprediction:whatthreshold touseforhit/miss?Varyandexplore?

✳ Howbesttoquantifyuncertainty intheskillscoreresultsbasedonvalidationsamplesize,uncertaintiesinobservations,andfromanyothersources.

Discussionquestions:Metrics

Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.Participationfromthecommunity:• Allpredictionmodelsandmethodsarewelcomefromtheworld-wideresearch• community(currently19methodsareregistered)• UserssubmittheirpredictionsforongoingCMEevents,listingtheirmethod• assumptionsandinputparameters• Researcherscanthenviewallofthepredictions,modelingdetails,andthe• ensembleaverageofallpredictedarrivaltimessubmittedbyparticipants

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

TheCMEscoreboardisaresearch-based forecastingmethodsvalidationactivitywhichprovidesacentrallocationforthecommunityto:

• submittheirforecastinreal-time• quicklyviewallforecastsatonceinreal-time• compareforecastingmethodswhentheeventhasarrived• viewtheaverageofallforecastsforeachevent(ensemble).

CMEArrivalTimeScoreboard

Communitypredictionsforthe5Nov2016CME

Pleasejoin!Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

CommunitypredictionsfortheJanuary7,2014CME(X1.2flare):

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

Averageofallsubmissions:12hoursearly,Kp geomagneticindex6to7.615submissions

Pleasejoin!Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.Therearecurrently19registeredmodels.

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboardAnyonecanviewpredictions,pleaseregistertosubmitpredictions.

BeginbyclickingAddPredictionunderthe"ActiveCMEs"sectionandselectyourforecasting"MethodType"fromthelist.Whileloggedin,ifyoudonotseeanyCMEs listedunderthe"ActiveCMEs"section,clickAddCMEtogetstarted.

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

Suggestedimprovementscomingsoon:• AutomaticforecastsubmissionviaanXMLfile• MailinglistthatnotifiesuserswhenanewCMEhasbeenaddedtothe

scoreboard• SeparategeomagneticstormscoreboardthatcanlinktoCMEscoreboard

Futureplans:• Showingdataintableinplotform• Automaticskillscorecalculations• QualityfactorforconfidenceinobservedICMEassociatedshockarrival• QualityfactorforconfidenceinlinkingobservedICMEarrivalwithCMEin

coronagraph• Yourideas?

CMEArrivalTimeScoreboard

http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard

Discussion:CMEarrivaltime&impactvalidationtechniques

CMEArrivalTimeErrorValidationExamples

CMEArrivalTimeErrorValidationExamples

0

20

40

60

80

100

0 20 40 60 80 100

Obs

erve

d R

elat

ive

Freq

uenc

y (%

)

Forecast Probability (%)

CME Arrival: Reliability Diagram of 52 Ensembles

line of

perfe

ct rel

iability

(5)

(4)

(4)

(2)

(7) (30)

Assessment:Confidence(likelihood)inCMEarrival

• ExamplereliabilitydiagramforCCMC/SWRCarrivaltimeforecasts

• Underforecasting intheforecastbinsbetween40-80%

• Slightlyoverforecasting inthe80-100%forecastbins

overforecasting

under-forecasting

NeedtoimproveconfidenceinCMEarrivalforecast:• ConsiderbetterwayoftranslatingCME“impactparameter”into

probabilitythattheCMEwillarrivewhichmoreaccuratelyrepresentshead-onvs.grazingimpacts(andtherangesinbetween)

Caution:smallSamplesizesinsomebins!!

LikelihoodofCMEarrivalforecastverification:BrierScore

AmethoddefiningthemeansquaredprobabilityforecasterrorsistheBrierScore:

N=numberofevents,pi =forecastprobabilityofoccurrenceforeventi,oi =1iftheeventwasobservedtooccurand0ifitdidnot.

Rangesfrom0to1,with0beingaperfectforecast.

UsingtheforecastprobabilityaboutthelikelihoodthattheCMEwillarrivesubmittedonthescoreboard.

TheBrierSkillScore(BSS)isthetheBrierscorerelativetoclimatology

Note:confidenceintervalsshouldbecomputedforverificationscores

LikelihoodofCMEarrivalforecastverification:Reliability

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4 5 6 7 8 9 10Pr

obab

ility

of R

ank

Rank

(b) CME Arrival Time: Rank Histogram of all Ensembles

Howwelldoestheensemblespreadrepresentthetruevariabilityoftheobservations?

TheU-shapedrankhistogramforsuggestsundervariability,indicatingthattheseensemblestonotsampleawideenoughspreadinCMEinputparameters.

EnsembleValidationSummary• Ensemble modeling gives a probabilistic forecast which includes an estimation of arrival time uncertainty from the spread in predictions and a forecast confidence in the likelihood of CME arrival.

• First results for 30 event sample: mean absolute arrival time error of 12.3 hours, RMSE of 13.9 hours, and mean error of -5.8 hours (early bias), comparable with other CME arrival time prediction errors reported in the literature.

• It was found that the correct rejection rate is 62%, and the false-alarm rate is 38%.

• Brier Score of 0.15 shows that the likelihood of CME arrival prediction is fairly accurate.

• However, the reliability diagram shows that the ensemble simulations are underforecastingthe likelihood that the CME will arrive in the forecast bins between 20-80%, and slightly overforecasting in the 1-20% and 80-100% forecast bins.

• For 8 out of 17 of the ensemble runs containing hits, the observed CME arrival was within the spread of ensemble arrival time predictions. The initial distribution of CME input parameters was shown to be an important influence on the accuracy of CME arrival time predictions. The rank histogram suggests undervariability in initial conditions; i.e., these ensembles do not sample a wide enough spread in CME input parameters.• The observed Kp was within ±1 of the predicted mean Kp for 11 out of 17 of the ensembles.

• Kp prediction errors: mean absolute error of 1.4, RMSE of 1.8, and mean error +0.4. • Overall tendency for the overprediction of Kp, for CME input speeds above ~1000 km/s.

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9

Prob

abilit

y of

Occ

urre

nce

(%)

KP

Probabilistic Kp Forecast Distribution (18 April 2014)90˚

135˚180˚

90˚-180˚ Kp isforecastusingENLILpredictedsolarwindquantitiesatEarthasinputtotheNewelletal.(2007)couplingfunction forthreeclockanglescenarios(ΘC=90°,135°,and180°)andallthreeanglescombined,assumingequallikelihood.

18April2014CME:DistributionofKp probabilityforecast

• ObservedKp:5duringperiod12:00-15:00UTon20April.• 84%oftheforecastsfallbetweenKp =5to7.ThemostlikelyforecastisforKp=7at41%,followedbyKp=5at27%andKp=6at16%likelihoodofoccurrence.• UsingthemeanKp forecastofKp=6,thepredictionerrorisKperror=Kppredicted- Kpobserved =1(overprediction)

Kp =5observed

Simulatedvs.ObservedCMEParametersØ Thedifferencefrom

differentobservationdatacanaffecttheresults.Forexample,thedifferenceofVmaxfromOMNIandACEis>200km/sfor3CMEs.ThecorrelationforNpmax isweakerifusingACE

Ø InseveralcaseswheretheCMEVmax isoverestimated,thereareinteractionsofmultipleCMEs

Ø Usingthefixedparameters(a6b1),theVmaxandNpmax areunderestimated.Theyareoverestimatedinthecaseofself-adjustedparameters(ace3b)

Ø SimilartrendsarefoundforthecorrelationsofmeanvaluesofCMEparameters.Themeantemperatureareoverestimatedinbothsettings FromLanJian