Scenario Planning for Sea Level Rise viaReinforcement Learning

Salman S. Shuvo, Yasin Yilmaz, Alan Bush, Mark Hafen

Abstract—Climate change and sea level rise impacts will affectcoastal communities with multiple threats, including increasedfrequency of compound events, such as storm surge combinedwith heavy precipitation. Accurately modeling how the stakehold-ers, such as governments and residents, may respond to sea levelrise scenarios (i.e., scenario planning) can assist in the creationof policies tailored to local impacts and resilience strategies. Inthis paper, our contributions are twofold. Firstly, considering asingle-agent model for government, we numerically show thatthe government’s policy on infrastructure improvement shouldbe based on the observed sea levels rather than the observedcost from nature. The latter refers to the straightforward policythat any responsive (but not proactive) government would follow.Through a reinforcement learning algorithm based on a Markovdecision process model we show that the precautionary measures,(i.e., infrastructure improvements triggered by the sea levels)are more effective in decreasing the expected cost than theaftermath measures triggered by the cost from nature. Secondly,to generate different scenarios we consider several sea levelrise projections by NOAA, and model different governmentand resident prototypes using cooperation indices in terms ofbeing responsive to the sea level rise problem. We present areinforcement learning algorithm to generate simulations for aset of scenarios defined by the NOAA projections and cooperationindices.

I. INTRODUCTION

Global climate change and its impacts, in terms of sealevel rise, have been extensively documented, analyzed andprojected [1]–[3]. Climate change and sea level rise impactswill affect coastal communities with multiple threats, includingincreased frequency of compound events - such as stormsurge combined with heavy precipitation [4]. This exacerbatessocial vulnerability, particularly in underserved communities[5], [6]; stresses coastal ecosystems [7], [8]; and impacts localeconomies by affecting property values, the tax base, and thecost of insurance, among other factors [9]. Because of sealevel rise, coastal communities are vulnerable to many of theseimpacts, and must build the adaptive capacity and resilienceframeworks to respond to these stressors through effectivedecision support and planning [10].

Overall, better information is needed for governments, plan-ners, coastal managers, and personnel in a variety of agenciesfor effective communication, decision making and adaptationplanning [10], [11]. This requires the participation of keyactors to communicate the science, the variability, and the riskof various scenarios to stakeholders [12]. Accurately modelinghow these agents may respond to sea level rise scenarioscan assist in the creation of policies tailored to local impacts

This work is supported by NSF award no. 1737598. Authors arewith the University of South Florida, Tampa, FL 33620, USA. (e-mail:yasiny@usf.edu).

and resilience strategies, and requires a variety of communityengagement and planning tools, including scenario planning[13]–[15].

Reinforcement learning (RL) provides a suitable theoreticalframework for generating agent-based scenarios [16]. The RLagent interacts with the environment by taking an action ateach time and receiving a cost/reward from the environmentin return. The objective of the agent is to minimize/maximizean expected sum of costs/rewards over time by choosingoptimal actions from an action set. At each time, as a result ofagent’s action, the system moves to a new state according toa probability distribution. The optimal policy for deciding onactions maps system states to actions, i.e., determines whichaction to take in which state [17].

In this paper, we consider a city setup with government asthe decision maker (i.e., RL agent), and nature and residents asthe environment which the agent interacts with. Our contribu-tions are twofold. Firstly, considering a single-agent model forgovernment, we numerically show that a rational government’spolicy on infrastructure improvement should be based on theobserved sea levels rather than the observed cost from nature.The latter is the straightforward policy that any responsive (butnot proactive) government would follow. Through simulationswe show that the precautionary measures, (i.e., infrastructureimprovements triggered by the sea levels) are more effectivein decreasing the total cost than the aftermath measures, (i.e.,infrastructure improvements triggered by the cost from nature).Secondly, to generate different scenarios we consider severalsea level projections by National Oceanic and AtmosphericAdministration (NOAA), and model different government andresident prototypes using cooperation indices in terms of beingresponsive to the sea level rise problem. The optimum policydepends on these cooperation indices, and can be found usingreinforcement learning techniques. We present a reinforcementlearning algorithm to generate simulations for a set of scenar-ios defined by the NOAA projections and cooperation indices.

The remainder of the paper is organized as follows. In Sec-tion II, the proposed RL model is explained. A reinforcementlearning algorithm for finding the optimal policy is presentedin Section III. Scenario simulations are given in Section IV,and the paper is concluded in V.

II. RL PROBLEM FORMULATION

We investigate the problem of when to invest in infras-tructure improvement against sea level rise, e.g., storm waterdrainage system, sea wall, levee, etc. Hence, at every timestep, e.g., a year, government makes a decision xn = 1(invest) or xn = 0 (no invest) for infrastructure improvement.

Fig.1. Proposed MDPmodel.

Accordingly,thecurrentstateofthecityinfrastructureagainstsealevelriseisgivenbysn =

nm=1 xm = sn 1+xn.

Denotingthesealevelriseintimeintervalnwithrn≥0thesealevelissimilarlygivenby n =

nm=1 rm = n 1+rn.

Wedefinethesystemstate S =(sn,n)asthepairofinfrastructurestateandsealevel, whichclearlysatisfiestheMarkovproperty: P(Sn|Sn 1,...,S0)=P(Sn|Sn 1).

Letusdenotethecostfromnatureateachtimestepwithzn,e.g.,thecostofflooding,stormsurge,hurricane,etc.Alsodenote with yn theresponseofresidentsateachtimestep.Inthis work, weuseabinaryresponseyn ∈{0,1}fortheresidentsconsideringtheresidents’decisiontosupport(yn =1)ornot(yn =0)thegovernment’sinvestmentforhandlingthesealevelriseproblem,e.g.,bypayinganextratax.Inthiswork,weinvestigategovernment’sdecisionundersealevelrisescenariosusingasingle-agent RL model.Inthiscontext,agentreferstothegovernment,andenvironmentreferstothenatureandresidentstogether. Theresponsesxn,yn,zn ofgovernment,residents,andnature,respectively,togetherdefinetheagent’scostcn=(2−yn)xn+zn ateachtimen.Thiscostisnormalizedbytakingtheinvestmentcost(2−yn)xn aunitcost withtheresidentsupport(twounitswithouttheresidentsupport).Thesecondcomponent zn isthenormalizedcostfromnaturewithrespecttotheinvestmentcost.Finally,thecumulativecostfunctionfortheagentisgivenby

CN =

N

n=0

ang[(2−yn)xn+zn], (1)

whichrepresentsthetotaldiscountedcostinNtimestepsfromnow.Thediscountfactorag ∈(0,1)determinestheweight(i.e.,importance)offuturecostsinagent’sdecisions.AsidefrombeingastandardparameterinRLcostfunction,ag hasanimportantcontextual meaninginthis work.Itrepresentshowmuchthegovernmentvaluesthefuturecostsduetothesealevelriseprobleminitsdecisionmakingprocess.Hence,wecallaggovernment’scooperationindex.

Theagent’sobjectiveisto minimizeE[CN]bychoosingitsactions{xn}overtime.Thisdefinesa MarkovDecisionProcess(MDP),assummarizedinFig.1.Everytimeagenttakesanactionxn,environmentreactstothatbyincurringacostcn.Consideringdiscretizedsealevelrisevalues̃rn ∈{0,1,2,...}thestatetransitiondiagramisgivenbyFig.2.Inourproblem,environmentconsistsofnatureandresidents.

Fig.2. MDPstatetransitiondiagram.

Fig.3. NOAAprojections(solidlines)forSt.Petersburg,FL[18],andourcurvefitting(dashedlines)fortheexpectationofGammadistribution.

Wenextdiscusssuitable modelsforthemtocompletetheproposed MDPmodel.

Tomodelnature’scostwestartwithfourdifferentNOAAprojectionsforsealevelriseinSt.Petersburg,FL[18],givenbyFig.3.Sincethesearesomeexpectedlevels,inoursimu-lationsweaccountforuncertaintyby modelingthesealevelrisevariableusingaGammadistribution,rn∼Gamma(α,β).Weset β = 0.5andα to matchE[n]withthe NOAAprojectioncurves.ThesuccessfulcurvefittingshownFig.3isobtainedbysettingα=5.2fixedforthelowprojection;byincreasingαfrom6.3to16.596with0.104incrementsfortheintermediate-lowprojection;byincreasingαfrom9.5to42.17with0.33incrementsfortheintermediate-highprojection;andbyincreasingαfrom14to69.44with 0.56incrementsforthehighprojection.Then, we modelnature’scostzn usingtheGeneralizedParetodistribution,whichiscommonlyusedto modelcatastrophiclosses[19].Theparametersettingsforzn aregivenas

zn∼GeneralizedPareto(k,σn,θ)

k=−0.001,θ=0,σn=η(n 1)a

(sn 1)b. (2)

Throughtheparametersη,a,bwecontroltheimpactofmostrecentsealeveln 1overnature’scostzn relativetothemost

recentinfrastructurestatesn 1.Residents’decisionyn is modeledusingBernoullidistri-

bution withprobabilityparameterdesignedthroughlogisticsigmoidfunction:

yn∼Bernoulli(pn),pn=σ(qn)=1

1+e(qn q0)

qn=n 1

m=1

an mr xmzm, (3)

wherethescoreqnreflectsthewillingnessofresidentstosharegovernment’sinvestmentcostsbasedonthecooperationindexar∈(0,1).Ittakesahighvalue,andyieldsahighprobabilityofsupportiftheresidents’cooperationindexishigh(ar≈1),nature’scosthasbeenseriousandthegovernmenthasbeenre-sponsiveespeciallyrecently.Ifatleastoneoftheseconditionsdonotexist,thenqn tendstogetsmallervalues,decreasingtheprobabilityofsupport.Anaveragevalueforqn isusedforthesigmoid’smidpointq0.

III.FINDINGTHEOPTIMALPOLICY

RLprovidesadata-drivensolutiontoMDPproblems.Ittyp-icallyupdatesavaluefunctionV(sn,n)=minxn E[CN|xn]inaniterativewaybasedontheBellmanequation

V(sn 1,n 1)=minxn

E[cn+V(sn,n)|xn,yn]

=min{zn+agV(sn 1,n),

2−yn+zn+agV(sn 1+1,n)}. (4)

ThevaluefunctionV(sn,n)definestheoptimalpolicy:

xn=1 if2−yn+agV(sn 1+1,n)<agV(sn 1,n),xn=0 otherwise.

(5)AlthoughthereareseveralRLalgorithms,ingeneraltheRLapproachlearnsthevaluefunctionbyexperiencingactionsandthecorrespondingcosts.In Algorithm1, weprovideanRLalgorithmbasedon Monte-Carlosimulationstolearntheoptimalgovernmentpolicyoninfrastructureinvestmentactions.

Algorithm1RLalgorithmforlearningoptimumpolicy

1:Input:ag,ar,Returns(s,):anarraytosavestates’returnsinalliterations;

2:Initialize:V(s,)← 0,∀s,;3:foriteration=0,1,2,...do4: Generateanepisode:Takeactionsusing(5)forN steps5: G(s,)← sumofdiscountedrewardsfrom(s,)till

terminationforallstatesappearingintheepisode;6: AppendG(s,)toReturns(s,);7: V(s,)← average(Returns(s,));8: ifV(s,)convergesforalls, then9: break

10: endif11:endfor

Algorithm1runsseveralepisodestoiterativelycomputethevaluefunctionV(s,)forallfeasiblestates.Eachepisodeis

Fig.4. ConvergenceofvaluefunctionV(s,)inAlgorithm1fortheNOAA’sintermediate-lowprojection.

a Monte-Carlosimulationinwhichseveralstatesarevisitedaccordingtothecurrentpolicydefinedbythecurrentvaluefunction.Attheendofeachepisodethevaluesofvisitedstatesareupdatedusingthereturn,i.e.,totaldiscountedcostfromastateuntiltermination,fromthesestates.Afterthestatevaluesconverge,thefinalstatevaluesareusedforgeneratingscenariosimulations,asdescribedinthenextsection.TheconvergenceofAlgorithm1isillustratedinFig.4.

IV.SCENARIOSIMULATIONS

Inthissection, wepresentsimulationresultsforseveralsealevelrisescenarios.To matchthecostduetohurricaneseachyearthatisreportedin[20],wechoosetheparametersη=2,a=0.4,b=0.5in(2)forthenature model. Weobtaindifferentscenariosbyvaryingthecooperationindicesagandarforthegovernmentandresidents,respectively,andbyconsideringdifferentNOAAprojectionsforsealevelrise.

Fig.4showstheconvergenceofthestatevaluesV(s,)inAlgorithm1consideringtheNOAAint-lowprojection.Foreachscenario,oncetheconvergedstatevaluesarefound,theresultantoptimalpolicyisusedtoassessthecostoftheRL-basedgovernment. Notethatthe RL-basedgovernmentisproactiveindealingwiththesealevelriseproblemasitmon-itorsthesealevelstatetogetherwiththeinfrastructurestate,andtakesprecautionary measuresbyimprovingtheinfras-tructurewhenevertheexpectedfuturecostofnotimprovingexceedstheimprovementcost.Considerareactive/responsivereal-worldgovernmentthatfollowsastraightforwardpolicybyimprovinginfrastructureafterexperiencingasignificantcostfromthenature.

InFig.5, wecomparetheoptimal RLpolicy withthisstraightforwardpolicyintermsofaveragetotalcostin100yearsforthelowandhighNOAAsealevelprojections.Itisseenthat withthesamenumberofinvestmentsonaveragetheproactivepolicythatactsaccordingtothesealevelandinfrastructurestateinsteadoftheultimatecostfromnaturegreatlyreducesthetotalcostforthegovernment.

Fig. 5. Optimum policy vs. straightforward policy in terms of average total cost in 100 years for the low (left) and high (right) NOAA sea level rise projections.

Fig. 6. Average total cost as a function of government cooperation index agfor different resident cooperation indices ar for NOAA high SLR projection.

Finally, in Fig. 6, we analyze the effect of cooperationindices. As expected, the average total cost decreases withgrowing cooperation index for both the government (ag) andresidents (ar). The cost is more than doubled if both thegovernment and residents are not cooperative (ag = ar = 0.1)compared to the full cooperative case (ag = 1, ar = 0.9).

V. CONCLUSIONS

In this paper, we presented a proactive government modelfor the sea level rise problem in a city environment consideringthe impacts from nature and residents. The proactive govern-ment, which learns the optimal infrastructure investment policy(yes or no at each time step) through reinforcement learningto minimize the expected economic cost over time, monitorsthe sea level state together with the infrastructure state, and

makes an infrastructure investment to alleviate the effectsof sea level rise problem whenever the expected future costof no investment exceeds the immediate cost of investment.This proactive strategy was shown to greatly outperform thestraightforward investment policy which improves the infras-tructure in the aftermath of a serious economic cost from thenature. We also demonstrated that the average total cost can besignificantly reduced as the government and residents becomemore cooperative in addressing the sea level rise problem.For the sea level rise amounts over time, different NOAAprojections are considered.

REFERENCES

[1] A. C. Hine, D. P. Chambers, T. D. Clayton, M. R. Hafen, andG. T. Mitchum, Sea Level Rise in Florida: Science, Impacts, andOptions. University Press of Florida, 2016. [Online]. Available:https://books.google.com/books?id=dSTNjwEACAAJ

[2] A. H. Sallenger Jr, K. S. Doran, and P. A. Howd, “Hotspot of acceleratedsea-level rise on the Atlantic coast of North America,” Nature ClimateChange, vol. 2, no. 12, pp. 884–888, 2012.

[3] C. J. Hapke, O. Brenner, R. Hehre, and B. J. Reynolds, “Assessinghazards along our Nation’s coasts,” US Geological Survey, Tech. Rep.,2013.

[4] T. Wahl, S. Jain, J. Bender, S. D. Meyers, and M. E. Luther, “Increasingrisk of compound flooding from storm surge and rainfall for major UScities,” Nature Climate Change, vol. 5, no. 12, p. 1093, 2015.

[5] S. B. Kashem, B. Wilson, and S. Van Zandt, “Planning for climateadaptation: Evaluating the changing patterns of social vulnerabilityand adaptation challenges in three coastal cities,” Journal of PlanningEducation and Research, vol. 36, no. 3, pp. 304–318, 2016.

[6] G. Schrock, E. M. Bassett, and J. Green, “Pursuing equity and justice ina changing climate: Assessing equity in local climate and sustainabilityplans in US cities,” Journal of Planning Education and Research,vol. 35, no. 3, pp. 282–295, 2015.

[7] R. J. Nicholls, “Planning for the impacts of sea level rise,” Oceanogra-phy, vol. 24, no. 2, pp. 144–157, 2011.

[8] T. Wahl, F. M. Calafat, and M. E. Luther, “Rapid changes in the seasonalsea level cycle along the US Gulf coast from the late 20th century,”Geophysical Research Letters, vol. 41, no. 2, pp. 491–498, 2014.

[9] X. Fu, J. Song, B. Sun, and Z.-R. Peng, “Living on the edge: Estimatingthe economic cost of sea level rise on coastal real estate in the TampaBay region, Florida,” Ocean & Coastal Management, vol. 133, pp. 11–17, 2016.

[10] T. Beatley, Planning for coastal resilience: best practices for calamitoustimes. Island Press, 2012.

[11] X. Fu, M. Gomaa, Y. Deng, and Z.-R. Peng, “Adaptation planning forsea level rise: A study of US coastal cities,” Journal of EnvironmentalPlanning and Management, vol. 60, no. 2, pp. 249–265, 2017.

[12] J. Tribbia and S. C. Moser, “More than information: what coastalmanagers need to plan for climate change,” Environmental science &policy, vol. 11, no. 4, pp. 315–328, 2008.

[13] P. R. Berke and M. R. Stevens, “Land use planning for climateadaptation: Theory and practice,” Journal of Planning Education andResearch, vol. 36, no. 3, pp. 283–289, 2016.

[14] A. Drogoul, “Agent-based modeling for multidisciplinary and participa-tory approaches to climate change adaptation planning,” in Proceedingsof RFCC-2015 Workshop, AIT, 2015.

[15] R. Potts, L. Jacka, and L. H. Yee, “Can we Catch em All’? Anexploration of the nexus between augmented reality games, urbanplanning and urban design,” Journal of Urban Design, vol. 22, no. 6,pp. 866–880, 2017.

[16] L. Panait and S. Luke, “Cooperative multi-agent learning: The state ofthe art,” Autonomous agents and multi-agent systems, vol. 11, no. 3, pp.387–434, 2005.

[17] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.MIT press, 2018.

[18] T. B. C. S. A. Panel, Recommended Projection ofSea Level Rise in the Tampa Bay Region. [On-line]. Available: http://www.tbrpc.org/wp-content/uploads/2019/01/SLR Recommendation OBRC 10-9-15.pdf

[19] C. Kleiber and S. Kotz, Statistical size distributions in economics andactuarial sciences. John Wiley & Sons, 2003, vol. 470.

[20] J. Weinkle, C. Landsea, D. Collins, R. Musulin, R. P. Crompton,P. J. Klotzbach, and R. Pielke, “Normalized hurricane damage in thecontinental United States 1900–2017,” Nature Sustainability, vol. 1,no. 12, p. 808, 2018.