Recent Trends in Skill for Some Leading Global NWP …...2019/05/07  · Recent Trends in Skill for...

RecentTrendsinSkillforSomeLeadingGlobalNWPCenters

RossNHoffmana,b,d,e,KrishnaKumarc,d,SidA.Boukabarad,KayoIdee,FanglinYangf,RobertAtlasa

7May2019

(a)NOAA/AtlanRcOceanographicandMeteorologicalLaboratory,Miami,Florida(b)CooperaRveInsRtuteforMarineandAtmosphericStudies,UniversityofMiami,Miami,Florida

(c)RiversideTechnologyInc.,CollegePark,Maryland(d)NOAA/NESDIS/STAR,CollegePark,Maryland

(e)UniversityofMaryland,CollegePark,CollegePark,Maryland(f)NOAA/NCEP/EnvironmentalModelingCenter,CollegePark,Maryland

IntroducRon

•  AlookatthedeterminisRcforecastsofthreeleadingNWPcenters(ECMWF,NCEP,UKMO)fortheyears2015-2017.

•  PAMs(primaryassessmentmetrics)suchasthe500-hPageopotenRalanomalycorrelaRon(AC)orthe250-hPawindRMSEareconvertedtoNAMs(normalizedassessmentmetrics)andthenaveragedintoSAMs(summaryassessmentmetrics).

01/10/19 TrendsinSkill 2

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

Upgrades2015-2017Center i Date Upgrade DeltaECMWF 1 20150512 IFSCycle41r1 2.10

2 20160308 IFSCycle41r2(CubicOctahedral1280) 1.313 20161122 IFSCycle43r1 2.584 20170717 IFSCycle43r3 5.22

NCEP 1 20150114 TIN14-46(T1534) -4.122 20160511 TIN16-11(4DEnVar) 7.373 20170719 SCN17-67(NEMSIO) 0.81

UKMO 1 20161121 PS38(satelliteobs.) 4.752 20170907 PS39(10-kmresoluRon) 2.82

01/10/19 TrendsinSkill 3

01/10/19 TrendsinSkill 4

120-h500-hP

aNHX

geo

potenR

alheightR

MSE(m

)

1998 2018

45

65

Context:20yearsofforecastskill

01/10/19 TrendsinSkill 5

Jan May Sep Jan May Sep Jan May Sep

79

11

ECMWFNCEPUKMO

ECMWFNCEPUKMO

|----------2015--------|---------2016--------|--------2017--------|

Win

d R

MSE

(m s

-1)

120-h500-hPaNHXvectorwindRMSE;MA(365)andMA(31)

01/10/19 TrendsinSkill 6

ScorecardsofIFSCycle45r1versusIFSCycle43r3.

FromECM

WFNew

leferNo.156

.

01/10/19 TrendsinSkill 7

ScorecardsofIFSCycle45r1versusIFSCycle43r3.FromECMWFNewleferNo.156.ShowingHRESvs.analysisonly.

01/10/19 TrendsinSkill 8

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

PAMstoNAMstoSAMs

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

PAMstoNAMstoSAMs•  WeogenfocusonafewkeyPAMs,butthismayignoreotherimportantaspects

offorecastskill.TheuseofSAMsincreasesstaRsRcalsignificanceandenablesexploringdifferentaspectsofforecastskill.

•  PAM/NAM/SAMdimension::coordinatevalues–  ForecastRme::24,48,72,96,120,144,168h–  Level::250,500,700,850,1000hPa–  Domain::northernhemisphereextratropics(NHX),southernhemisphereextratropics(SHX),

tropics–  Variable::height(Z),temperature(T),wind(V)–  StaRsRc::anomalycorrelaRon(AC),rootmeansquareerror(RMSE),absolutemeanerror(AME,

theabsolutevalueofbias)–  VerificaRonRme::every24hat0000UTCduring2015-2017–  Center::ECMWF,NCEP,UKMO

•  ReferencesamplefornormalizaRon–  All::(verificaRonRme,center)–  ByCenter::(verificaRonRme)

01/10/19 TrendsinSkill 9

01/10/19 TrendsinSkillWind RMSE (m s-1)2 5 10 20

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

24 h48 h72 h96 h120 h144 h168 hAllECMWFJan

10

NAM

/ EC

DF

500-hPaNHXvectorwindRMSEECDFnormalizaRonfuncRons

01/10/19 TrendsinSkill 11Wind RMSE (m s-1)6 7 8 9 10 11 12

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

8

8

8

8

8

8

9

9

9

9

9

O

O

O

O

O

O

N

N

N

N

N

D

D

D

D

D

D

1

1

1

1

1

1

2

2

2

2

2

3

3

3

3

3

3

4

4

4

4

4

5

5

5

5

5

5

6

6

6

6

6

7

7

7

7

7

7

1

D

AllECMWFNCEPUKMOJan. . .Dec

NAM

/ EC

DF

120-h500-hPaNHXvectorwindRMSEECDFnormalizaRonfuncRons

01/10/19 TrendsinSkill 12

Example

0.65 0.70 0.75 0.80 0.85 0.90 0.95

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

2polar3pgps

3polar

cntrl

cntrl3polar3pgps2polar

0.0 0.1 0.2 0.3 0.4 0.5 0.6

01

23

45

6

●

●●●

●●

●

●

●●●

●●

●

●

●●●

●●

●

●●

●

maxavgminstep

01/10/19 TrendsinSkill 13

x = PAM

s =

n ×

NAM

•  ECDFnormalizaRon– NAM=(rank(PAM)-1/2)/n– RankrelaRvetoref.sample

•  Forminmax,andothernormalizaRons– NAM=aPAM+b– a,bdependonref.sample

ECDFNAMiscalculatedfromtherank

01/10/19 TrendsinSkill 14

Jan May Sep Jan May Sep Jan May Sep

0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO

|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;All;MA(365)andMA(31)

Jan May Sep Jan May Sep Jan May Sep

0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 15

|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;ByCenter;MA(365)andMA(31)

Jan May Sep Jan May Sep Jan May Sep

0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 16

|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;ByMonth;MA(365)andMA(31)

020000

60000

100000

0 0.2 0.4 0.6 0.8 1

050000

150000

250000

0 0.2 0.4 0.6 0.8 1

UKMONCEPECMWF

050000

100000

150000

−0.5 −0.1 0.3 0.7 1.1 1.5

01/10/19 TrendsinSkill 17

Cou

nts

a cb

NAM

020000

60000

100000

0 0.2 0.4 0.6 0.8 1

050000

150000

250000

0 0.2 0.4 0.6 0.8 1

UKMONCEPECMWF

050000

100000

150000

−0.5 −0.1 0.3 0.7 1.1 1.5

HistogramsforallNAMsfor(a)forECDF,(b)forminmax,and(c)forrescaled-minmaxnormalizaRons

01/10/19 TrendsinSkill 18

SAM

ECMWF NCEP UKMO

0.4

0.5

0.6

ECMWFNCEPUKMO

2015 2016 20170.25

0.50

0.75

ECMWFNCEPUKMO

Center Year

24 48 72 96 120 144 168

0.25

0.50

0.75

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 19

24 48 72 96 120 144 1680.25

0.50

0.75

ECMWFNCEPUKMO

201520162017

SAM

Forecast time (h)Forecast time (h)

01/10/19 TrendsinSkill 20

Z T V

0.25

0.50

0.75

ECMWFNCEPUKMO

NHX SHX Tropics0.25

0.50

0.75

ECMWFNCEPUKMO

SAM

Variable Domain

NHX SHX Tropics

0.25

0.50

0.75

ECMWFNCEPUKMO

24 48 72 96 120 144 1680.25

0.50

0.75

ECMWFNCEPUKMO

NHXSHXTropics

01/10/19 TrendsinSkill 21

SAM

Forecast time (h)Domain

AC RMSE AME0.25

0.50

0.75

ECMWFNCEPUKMO

250 500 700 850 1000

0.25

0.50

0.75

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 22

SAM

Level (hPa) Statistic

ByCenternormalizaRon•  ReferencesamplefornormalizaRon

–  All::(verificaRonRme,center)–  ByCenter::(verificaRonRme)

01/10/19 TrendsinSkill 23

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

24 48 72 96 120 144 1680.4

0.5

0.6

ECMWFNCEPUKMO

201520162017

2015 2016 2017

0.4

0.5

0.6

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 24

SAM

Forecast time (h)Year

Upgrades2015-2017Center i Date Upgrade DeltaECMWF 1 20150512 IFSCycle41r1 2.10

2 20160308 IFSCycle41r2(CubicOctahedral1280) 1.313 20161122 IFSCycle43r1 2.584 20170717 IFSCycle43r3 5.22

NCEP 1 20150114 TIN14-46(T1534) -4.122 20160511 TIN16-11(4DEnVar) 7.373 20170719 SCN17-67(NEMSIO) 0.81

UKMO 1 20161121 PS38(satelliteobs.) 4.752 20170907 PS39(10-kmresoluRon) 2.82

01/10/19 TrendsinSkill 25

day-by-day

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

1 2 3 4

01/10/19 TrendsinSkill 26

SAM

|----------2015--------|---------2016--------|--------2017--------|

ECMWF

day-by-day

Upgrades2015-2017Center i Date Upgrade DeltaECMWF 1 20150512 IFSCycle41r1 2.10

2 20160308 IFSCycle41r2(CubicOctahedral1280) 1.313 20161122 IFSCycle43r1 2.584 20170717 IFSCycle43r3 5.22

NCEP 1 20150114 TIN14-46(T1534) -4.122 20160511 TIN16-11(4DEnVar) 7.373 20170719 SCN17-67(NEMSIO) 0.81

UKMO 1 20161121 PS38(satelliteobs.) 4.752 20170907 PS39(10-kmresoluRon) 2.82

01/10/19 TrendsinSkill 27

day-by-day

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

1 2 3

01/10/19 TrendsinSkill 28

SAM

|----------2015--------|---------2016--------|--------2017--------|

NCEP

day-by-day

Upgrades2015-2017Center i Date Upgrade DeltaECMWF 1 20150512 IFSCycle41r1 2.10

2 20160308 IFSCycle41r2(CubicOctahedral1280) 1.313 20161122 IFSCycle43r1 2.584 20170717 IFSCycle43r3 5.22

NCEP 1 20150114 TIN14-46(T1534) -4.122 20160511 TIN16-11(4DEnVar) 7.373 20170719 SCN17-67(NEMSIO) 0.81

UKMO 1 20161121 PS38(satelliteobs.) 4.752 20170907 PS39(10-kmresoluRon) 2.82

01/10/19 TrendsinSkill 29

day-by-day

01/10/19 TrendsinSkill 30

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

1 2

SAM

|----------2015--------|---------2016--------|--------2017--------|

UKMO

day-by-day

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

01/10/19 TrendsinSkill 31

SAM

|----------2015--------|---------2016--------|--------2017--------|

Average

day-by-day

Jan May Sep Jan May Sep Jan May Sep

0.4

0.5

0.6

ECMWFNCEPUKMO

ECMWFNCEPUKMO

01/10/19 TrendsinSkill 32

SAM

|----------2015--------|---------2016--------|--------2017--------|

MA(365)andMA(31)

01/10/19 TrendsinSkill 33

SAM

MonthJan Mar May Jul Sep Nov

0.4

0.5

0.6

●

●

● ●

●

●●

●

●

●

●

●

● ECMWFNCEPUKMO

Summary

•  Allthreecentersimproveoverthethreeyearperiod.NCEPshort-termforecastskillsubstanRallyincreasesduringtheperiod.

•  SAMsindicatethatintermsofforecastskillECMWFisbeferthanNCEP,whichisbeferthanbutapproximatelythesameasUKMO.

01/10/19 TrendsinSkill 34

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

•  However,theobservedimpactsarewithinthecontextofslowlyimprovingforecastskillforoperaRonalglobalNWPascomparedtoearlieryears.

•  TheuseofSAMsimprovesthesignaltonoiseraRoandclearimprovementsinSAMarerelatedtotheECMWFJuly2017upgradetoIFSCycle43r3,theNCEPMay2016replacementofthe3DEnVarwiththe4DEnVar,andtheUKMONovember2016(PS38)introducRonofimproveduseofsatelliteobservaRons.

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

Concludingremarks

•  WeogenfocusonafewkeyPAMs,butthismayignoreotherimportantaspectsofforecastskill.TheuseofSAMsincreasesstaRsRcalsignificanceandenablesexploringdifferentaspectsofforecastskill.

•  ClearlythesystemslaggingECMWFcanimprove,andthereisevidencefromSAMsinaddiRontothe4DEnVarexamplethatimprovementsinforecastanddataassimilaRonsystemsaresRllleadingtoforecastskillimprovements.

•  Infuturework,itmightbeinteresRngtoincludeothercentersandtoaddPAMsforrelaRvehumidityandprecipitaRon,forecastvariablesforwhichthereiscurrentlymajorroomforimprovement.

01/10/19 TrendsinSkill 35

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

more...•  email:•  ross.n.hoffman@noaa.gov

•  Dec2018WAFpaper:•  doi:10.1175/WAF-D-18-0117.1

•  AMSpresentaRon:•  hfps://ams.confex.com/ams/2019Annual/meeRngapp.cgi/Paper/

350739

01/10/19 TrendsinSkill 36

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

ADetecRveStory•  Webeganwith5centers:

CMC,ECMWF,FNMOC,NCEP,UKMO

•  Wekeptthe3best,partlybecauseFNMOCresultsdidnotmakesense

•  Colorshavechangedfromthe3centerscase.

01/10/19 TrendsinSkill 37

CMC ECMWF FNMOC NCEP UKMO0.25

0.50

0.75

CMCECMWFFNMOCNCEPUKMO

24 48 72 96 120 144 1680.4

0.5

0.6

● ●● ● ● ● ●

● ● ● ●● ● ●

●

●

●● ● ● ●

●

●●

● ●● ●

●

●

●● ● ●

●

● ● ● ● ● ●●

●

●

●● ● ● ●

●●

● ● ● ● ●

●

●● ● ● ●

●●

● ● ●●

●

●

CMCECMWFFNMOCNCEPUKMO

201520162017

2015 2016 2017

0.4

0.5

0.6

CMCECMWFFNMOCNCEPUKMO

01/10/19 TrendsinSkill 38

SAM

Forecast time (h)Year

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

1 2

01/10/19 TrendsinSkill 39

SAM

|----------2015--------|---------2016--------|--------2017--------|

CMC

day-by-day

Jan May Sep Jan May Sep Jan May Sep

0.3

0.5

0.7

1 2

01/10/19 TrendsinSkill 40

SAM

|----------2015--------|---------2016--------|--------2017--------|

FNMOC

day-by-day

year

SAM

2015 2016 2017

0.4

0.5

0.6

| arraytype = sam | normalization = ecdf | refsample = allNWP | gamma = 0.2652 |SAM[year,exp], only for (var=V;lev=1000), averaged over (stat,ftime,domain,day,month)

p = 0.95 c.i. shaded in grey

CMCECMWFFNMOCNCEPUKMO

outlines are for allNWP rescaled SAM

year

SAM

2015 2016 2017

0.4

0.5

0.6

| arraytype = sam | normalization = ecdf | refsample = allNWP | gamma = 0.1507 |SAM[year,exp], without (lev=1000), only for (var=V), averaged over (stat,ftime,domain,lev,day,month)

p = 0.95 c.i. shaded in grey

CMCECMWFFNMOCNCEPUKMO

outlines are for allNWP rescaled SAM

01/10/19 TrendsinSkill 41

SAM

Year Year

Wind (250,500,700,850) Wind (1000)

Why?•  Inmid2016,theNAVGEMgridssharedwithNCEPchangedfrom1

degreeto1/2degreelaRtude-longitude.•  Becauseofthewaythefieldsarefiltered,thischangemakesit

seemlikeNAVGEMforecastskillisdegradinginourassessmentsusingtheVSDBstaRsRcs.–  MostoftheNAVGEMfieldsusedinourassessmentsarefilteredwiththe

same2done-passShapirosmootherde-smootherappliedingridpointspaceforboth1and1/2degreefields.

–  Asaresultthe1/2degreefieldshavemoreenergypresentintheinherentlyhardtopredictsmallestscales,resulRnginanapparentdropinforecastskill.

–  DuringVSDBprocessingnofurtherfilteringisapplied.

01/10/19 TrendsinSkill 42

Butwhataboutlowlevelwinds?•  FilteringisappliedtoallgeopotenRalheights,alltemperatures,andallwindsabove900hPa.– Thusinourstudy,only1000hPawindswerenotaffectedbythischange.

•  ThankstoElizabethSaferfield/NRL-MontereyandRandalPauley/FNMOCforhelpinunravelingthispuzzle.

01/10/19 TrendsinSkill 43

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

more...•  email:•  ross.n.hoffman@noaa.gov

•  Dec2018WAFpaper:•  doi:10.1175/WAF-D-18-0117.1

•  AMSpresentaRon:•  hfps://ams.confex.com/ams/2019Annual/meeRngapp.cgi/Paper/

350739

01/10/19 TrendsinSkill 44

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

ApplicaRontoimpactexperiments•  TheoriginoftheSAMworkwastosummarizeOSEresultsforadatagapimactstudy.

•  WerepeatedtheimpactstudyinsimulaRon(OSSEmode)tovalidateourOSSEsystem(CGOP).

•  IdeallyeachOSSEcomponentmustberealisRc,howevertheseOSSEsused–  LowerresoluRonGDAS/GFS– NoobservaRonerrors

01/10/19 TrendsinSkill 45

ObservingSystemSimulaRonExperiments

•  OSSEshavebeenusedsincethe1950sto–  Evaluateobservingsystemsintermsofaccuracyandcoverage(e.g.,inplanningFGGE)

–  GuidedecisionmakerstoallocateresourcestomiRgatecostsandleadRmeinreality

–  Conducttradestudiesofinstrumentsandsystems,and–  DesignandtestnewDAmethods

01/10/19 TrendsinSkill 46

OSSE System

NR Simulate obs.

DA Forecasts

Future OSSEs 23 April 22, 2015

Revised: October 23, 2014

CalibrationVerification

ExperimentalSetup:DataGapScenario•  Inter-comparison:OSSEvsOSEforControl,3Polar,2Polar

[OriginalOSEworkbyBoukabaraetal(2016b)]

Period 7Julyto7August,2014

Obssystemconfig.

2014:allconvenRonal+satellitedatagapscenarios

OSSEonly:perfectobs

DAS 3DEnVarwithT670/T254n=80ensemble

Forecast 0000UTCdailyupto168h

VerificaRon OSSE/OSE:owncntrlanalysis

OSSEsystemvalidaRon/calibraRon•  BeforeapplyanOSSEsystemtoanewproposedsensor,wewanttocheckiftheresultsforadatadenialexperimentmatch

•  Ifnot(andogenOSSEresultsareoverlyopRmisRc)wemustcalibrateourOSSEsystemby–  TuningtheobservaRonerrors;and/or–  ChangingsomeparameterizaRonsintheforecastmodel;or–  AdjusRngtheOSSEresultsagerthefact

•  ForthesimulaRonexperimentswith“perfect”observaRons,theOSSEwasoverlyopRmisRc,butnotintermsofSAMs!

01/10/19 TrendsinSkill 48

Results:ForecastSkill(PAM-AC&RMSE)

01/10/19 TrendsinSkill 49

•  Alldatagapscenariosresultinpoorforecastskills•  TendencyofimpactmostlyasexpectedalthoughtherearebitofvariabiliResinOSSEvsOSEinter-comparisonresults

00 24 48 72 96 120 144 168

02

46

810

●

●

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

00 24 48 72 96 120 144 168

0.0

0.5

1.0

1.5

2.0

2.5

●

●

3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

●

●

●

● ●● ● ●

●

●●

● ● ● ●●

00 24 48 72 96 120 144 168

0.5

0.6

0.7

0.8

0.9

1.0

●

●

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

● ● ●●

●

●

●

●

● ●●

●

●

●

●

●

00 24 48 72 96 120 144 168

−0.04

0.00

0.02

0.04

●

●

3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

● ● ● ● ●●

●

●

● ●●

●

●

●

●

●

Forecast time (h)

V 250 hPa Tropics RMSE (m s-1)Z 500 hPa NHX AC

Forecast time (h)

Results:SAMsglobalandvs.forecastRme

01/10/19 TrendsinSkill 50

cntrl−OSSE 3polar−OSSE 2polar−OSSE

0.3

0.5

0.7

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

00 24 48 72 96 120 144 1680.1

0.3

0.5

0.7

0.9

●

●

●●

● ● ● ●

●

●

●● ● ● ● ●

●

●

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

a bSA

M

Forecast time (h)Experiment

cntrl-OSSE

cntrl-OSE

3polar-OSE

2polar-OSE

3polar-OSSE

2polar-OSSE

Results:SAMsbylevelanddomain

01/10/19 TrendsinSkill 51

250 500 700 850 1000

0.3

0.5

0.7

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

NHX SHX Tropics

0.3

0.5

0.7

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

a bSA

M

DomainLevel (hPa)

more...•  email:•  ross.n.hoffman@noaa.gov

•  Oct2018JTECHpaper:•  doi:10.1175/JTECH-D-18-0061.1

•  AMSpresentaRon:•  hfps://ams.confex.com/ams/2019Annual/meeRngapp.cgi/Paper/

353842

01/10/19 TrendsinSkill 52

cntrl−OSSE 3polar−OSSE 2polar−OSSE

0.3

0.5

0.7

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

Experiment

cntrl-OSSE

cntrl-OSE

3polar-OSE

2polar-OSE

3polar-OSSE

2polar-OSSE

Control3Polar2Polar