View
5
Download
0
Category
Preview:
Citation preview
ProjectName:HighFrequencyPhysics-BasedEarthquakeSystemSimulations(Year1of2)EndofYearSummary
PINAME:ThomasH.JordanCo-PI(s):JacoboBielak,CarnegieMellonUniversity,
PoChen,UniversityofWyoming,YifengCui,SanDiegoSupercomputerCenter,
PhilipMaechling,SouthernCaliforniaEarthquakeCenter,KimOlsen,SanDiegoStateUniversity,RicardoTaborda,UniversityofMemphis
PROGRAM:INCITEALCFYEAR:2015
PerformancePeriod:January1-December31,2015(Q1,Q2,Q3,Q4)
ALCFProjectName:GMSeismicSimOLCFProjectName:GEO112
ProjectUsagePleasecommentontheuseofyourMiraallocation.Forexample,thiscouldincludeperiodsoflowutilizationduetocode/modeldevelopment,periodsofhighutilizationduetosignificantcampaigns,significantuseofthebackfillqueueortheINCITEoverburnpolicy.Ifmorethan50%ofyourallocationwasusedupbyjobsbelowthecapabilitysize(lessthan8-racks),pleasecommentonthetechnicalbarrierspreventingyourworkloadorcodefromscalingto8-racksandbeyond.Whileour2015Mirausageproducedimportantscientificresults,ourusagepatternwasnotoptimal.Oneissueisthatweusedour2015ALCFallocationinthefirstsixmonthsoftheyear.Then,wecontinuedtorunjobsataslowerrate,usingtheback-fillqueue,duringtheremainderoftheyear.WewillplanforamoreconsistentusageofALCFcomputingtimeduring2016toavoidrunningoutofallocationhoursonMiraduring2016.Anotherissueweareworkingtoaddressisthesizeofourroutinejobs.OurbestresultsfromMirain2015arefromouriterative,data-intensive,full3Dtomographicimprovementsto3DseismicvelocitymodelsforCalifornia.ThestandardcalculationsinthisprocessonMiraarenow1rackinsize.However,in2016weexpecttoruntomographicinversionsvaliduptohigherfrequencies,whichwillrequireatleastafactorof8increaseincomputationalscalefortheseroutinejobs.However,thedatastoragerequirementsforourinversionswillalsoincreasesubstantiallywhenwemovetohigherfrequencies.WearecurrentlyevaluatingdatacompressiontechniquesforuseonourALCFdatathatwillreducetherequirementsofthesehigherfrequencyinversions.WehaveidentifieddatacompressionevaluationasprioritywithourALCFCatalystfor2016.
ReportonProjectMilestones• Pleaseinsertacopyofthemilestonesasoutlinedintheproposal• Provideastatusoneachofthemilestones• Listanyadditionalmajoraccomplishmentsthusfarthisyear.Pleaseinclude
scientificandcomputationaldetailsofsimulationsundertaken,includingimagesifpossible
• Listanycommentstoclarifythestatusoftheprojecto Forexample,fundingchallengesslowedwork,changeofsciencescope
withanexplanation,etc DuringthisINCITEallocation,researchersfromtheSouthernCaliforniaEarthquakeCenter(SCEC)usedcomputingtimeandstoragespaceonALCFandOLCFINCITEcomputationalresourcestocarryoutresearchonhigh-frequencyphysics-basedearthquakesystemsimulations.Whilethis2015end-of-yearreportisrequestedbyALCF,wedescribetheoverallprogressinourINCITE-supportedresearchprogramusingbothOLCFandALCFresources.Wherepossible,weidentifywhichINCITEresourceswereusedtoproducespecificresults.In2015,SCECusedINCITEresourcestoinvestigateabroadrangeofscientificissues.Our2015-2016INCITEproposaldefinedthefollowingfourobjectives:
• O1:Improvetheresolutionofdynamicrupturesimulationsbyanorderofmagnitudeandinvestigatetheeffectsofrealisticfrictionlaws,geologicheterogeneity,andnear-faultstressstatesonseismicradiation.
• O2:Extenddeterministicsimulationsofstronggroundmotionsto10Hzforinvestigatingtheupperfrequencylimitofdeterministicground-motionprediction.
• O3:Computephysics-basedProbabilisticSeismicHazardAttenuation(PSHA)mapsandvalidatethoseusingseismicandpaleo-seismicdata.
• O4:Improve3Dearthstructuremodelsthroughfull3Dtomographyusingobservedseismicityandambientnoise.
Duringthefirstyearofourcurrentallocation,SCECresearchershaveprogressedtowardseachoftheseobjectives:WedefinedandimprovedanewcentralCalifornia3Dvelocitymodelusingfull3Dtomography(O4).Weproducedacomprehensive,physics-basedhazardmodelfortheLosAngelesregionvaliduptoseismicfrequenciesof1Hz(O3),weperformedhigh-frequencysimulations(upto8Hz)onTitanusingGPU-optimizedfinite-differencecodes(O2)whileincorporatingnewaspectsofearthquakephysics(O1).Here,weprovideaprogressupdateonthemilestoneswedefinedinour2015INCITEallocationrequest,followedbyasummaryofourmostsignificantaccomplishments.Theprogressmadein2015highlightsonceagainhowthispartnershipallowsustoperformthemostscientificallyadvancedearthquakegroundmotionandseismichazardcomputationsconductedworldwide.
SummaryofMilestonesProgress:Weidentifiedthefollowingeightmilestones[M1–M8]inouroriginalINCITEallocationrequest.Abriefsummaryofprogressispresentedbelow.
Year1MilestoneDescriptions MilestoneAchievementStatusM1 Usefull3Dtomographyandcomparative
validationsusingtoimproveexistingCaliforniavelocitymodelsforuseinhighfrequencywavepropagationsimulationsat0.2Hz
Achieved.UsedMiratocalculatesixiterationsofCentralCaliforniaModel.Theimproved3DmodelisnowavailabletogroundmotionmodelersthroughSCECdistributionsoftwarecalledUCVM.
M2 Runhighfrequencyforwardsimulationsusingalternativematerialattenuation(Q)andseismicvelocitymodels(CVMs).Comparetheimpactofmaterialproperties,topography,andmodelsincludingspatialvariability(heterogeneities)andsoft-soildeposits(orgeotechnicallayers)on4Hz+simulationsbysimulatingforwardeventsusingalternativemodelsandcomparingresultsamongsyntheticsandwithdata.
Started,notcompleted.AWP-ODCandHerculescodebrancheshavetestedthesephysics.Currentlyrunningbaseline4Hzsimulationswithoutthesephysicswithgoodagreementat4Hzamong3wavepropagationcodeswithasimplevelocitystructure.
M3 Runhighfrequencyforwardsimulationsusingalternativeapproachestoincludetheeffectsofoff-faultandnear-surfaceplasticdeformation.Comparetheimpactofalternativeplasticitymodels(linear-equivalent,3D+1Dhybrid,full3Dplastic)on4Hz+simulationsbysimulatingforwardeventsandcomparingtheresultsamongsyntheticsandwithempiricalrelationshipsanddata.
Started,notcompleted.AWP-ODCandHerculescodebrancheshavetestedthesephysics.Currentlyrunningbaseline4Hzsimulationswithoutthesephysicswithgoodagreementat4Hzamong3wavepropagationcodeswithasimplevelocitystructure.
M4 Calculatea1.0HzCyberShakeHazardcurve.UseupdatedCVMs,sourcemodels,andcodestocalculateahigherfrequencyCyberShakehazardcurve
Achieved.UsedTitanandBlueWaterstocalculateaCyberShake1HzLosAngelesareaprobabilisticseismichazardmodelbasedon336site-specifichazardcurves.
Year2MilestoneDescriptions ObjectiveM5 Usefull3Dtomographyandcomparative
validationsusingtoimproveexistingCaliforniavelocitymodelsforuseinhighfrequencywavepropagationsimulationsat0.5Hz
Notstarted
M6 Runhighfrequencyforwardsimulationsusingalternativematerialattenuation(Q)andseismicvelocitymodels(CVMs).Comparetheimpactofmaterialproperties,topography,andmodels
Notstarted
includingspatialvariability(heterogeneities)andsoft-soildeposits(orgeotechnicallayers)on8Hz+simulationsbysimulatingforwardeventsusingalternativevelocitymodelsandcomparingtheresults.
M7 Runhighfrequencyforwardsimulationsusingalternativeapproachestoincludetheeffectsofoff-faultandnear-surfaceplasticdeformation.Comparetheimpactofalternativeplasticitymodels(linear-equivalent,3D+1Dhybrid,full3Dplastic)on8Hz+simulationsbysimulatingforwardeventsandcomparingtheresultsamongsyntheticsandwithempiricalrelationshipsanddata.
Notstarted
M8 Calculatea1.5HzCyberShakeHazardcurve.UseupdatedCVMs,sourcemodels,andcodestocalculateahigherfrequencyCyberShakehazardcurve
Notstarted
MajorProjectAccomplishments:1)Wecompleteda1-HzurbanseismichazardmodelfortheLosAngelesregion(OLCFwithproductsfromALCF).SCEC'sresearchteamusedtheOLCFTitanandNCSABlueWaterssupercomputerstoperformCyberShakeStudy15.4.ThiscomputationdoubledthemaximumseismicfrequencyrepresentedintheLosAngelesurbanseismichazardmodel,from0.5Hzto1Hz(Figure1).Seismichazardcurveswerederivedfromlargeensemblesofseismogramsatfrequenciesbelowthismaximumfor336surfacesitesdistributedacrosstheLosAngelesregion.Thisnewprobabilisticmodelusesrefinedearthquakerupturedescriptionsthroughrevisionstotheconditionalhypocenterdistributionsandtheconditionalslipdistributions.ThisseismichazardcalculationusedtheCVM-S4.26.GTL3Dvelocitymodel,whichwasvalidatedandimprovedusingALCFMira,asthebestavailablesouthernCalifornia3Dvelocitymodel.TheCS15.4modelprovidesnewseismichazardinformationofinteresttobroadimpactcustomersofCyberShake,includingseismologists,utilitycompanies,andcivilengineersresponsibleforCaliforniabuildingcodes.Thenewmodel,whichsamplesthecompleteUniformCaliforniaEarthquakeRuptureForecast,willberegisteredintotheUSGSUrbanSeismicHazardMappingProject(http://earthquake.usgs.gov/hazards/products/urban/).
TheGPU-basedanelasticwavepropagationAWP-ODCsoftwarewascoupledwiththeCPU-basedpost-processingcalculationsthatsynthesizedover300millionseismograms.InStudy15.4,SCECutilizedapproximately200pilotjobstorunCyberShaketasksonTitanresources.Over80%ofthenode-hoursburnedonTitanwerefromjobswhichranon25%ormoreofthemachine.Approximately200TBofSGTdatawastransferredfromTitantoBlueWatersautomaticallyaspartoftheworkflow.OnTitan,theacceleratedcalculationsoftheGPUStrainGreenTensor(SGT)implementationis6.3timesmoreefficientthantheCPUimplementation,comparedtheperformanceonXK7toXE6atnode-to-nodelevel,whichsavedusmorethansixtymillionofcore-hoursoverthecourseofthestudy.OurGPUdevelopmentwasrecognizedwithNVIDIA’s2015GlobalImpactAward.“Thefullthree-dimensionaltreatmentofseismic-wavepropagationhasthepotentialtoimproveseismichazardanalysismodelsconsiderably,andthatiswheretheacceleratingtechnologyisparticularlyhelpfulatthismoment,”saidThomasJordan,directorofSCEC.“WithGPUcomputingpowerwe’regaininginsightastohowthegroundwillmoveinhigh-riskareas,andhowwecanbetterplanfortheaftermathofamajorevent.”http://www.hpcwire.com/off-the-wire/sdsc-researchers-awarded-nvidia-2015-global-impact-award/2)WerefinedthecentralCalifornia3Dvelocitymodel,thatwecallCentralCaliforniaArea(CCA),andusedfull3Dtomographycomputationalmethodsto
validateandimprovethe3Dseismicvelocitymodelusingbothobservedmoderateearthquakesandambientseismicnoiseobservations(ALCF).Wehavefurtherimprovedthecomputationalefficiencyofourfull3Dtomography(F3DT)workflowonALCFMiraandarenowapplyingittoCentralCaliforniaandstatewide.Anaccurate3Dvelocitymodelisessentialinputneededforaccuratedeterministicearthquakewavepropagationsimulations.GradualimprovementsinourvelocitymodelshaveallowedustoincorporateanincreasingvolumeofobservedseismogramsintoourF3DTworkflow,whichisallowingustoresolvefinerstructuraldetailswithhigheraccuracy.Figure2belowshowsthestudyareaandresultsfromthe6thiteration.Thestartingmodelisdefinedusinga500mgridspacingandweusetrilinearinterpolationinbetweenthegridpointswhenconstructingmeshes.Themodelcoversdepthsdownto50km.Wehavealsointegratedalossy,onlinecompressioncodezfpintothefull-3Dtomographybasedonthescattering-integralmethod(F3DT-SI)workflow.ThelastF3DTimplementation,inparticularthescattering-integralimplementation(F3DT-SI),requiredhighdiskstoragecostandtheassociatedI/Ooverheadofarchivingthe4Dspace-timewavefieldsofthereceiver-orsource-sidestraintensors.WehavesuccessfullyintegratedalossycompressionalgorithmintoourF3DT-SIworkflowtosignificantlyreducethediskspaceforstoringthesedata.
Figure2.Shearwave(Swave)velocityat(top)2km,(middle)10km,and(bottom)20kmdepthsin(left)theinitialmodelCCA00,(middle)the6thiterationmodelCCA06,and(right)theperturbations.ThecolorbaronthelowerleftcornerofeachplotshowstherangeofthecolorscalewithredindicatingrelativelyslowSwavevelocitiesandblueindicatingrelativelyfastSwavevelocities.Blacksolidlinesshowmajorfaultsinourstudyarea.3)Weperformedhigh-frequency(upto8Hz)simulations(OLCFTitan)TheSCECresearchersranhighfrequencygroundmotionsimulationsonTitanusingGPU-enabledfinitedifferentwavepropagationsolvers,andafinite-elementwavepropagationsolver,Hercules,whichintegratesanefficientoctree-basedhexahedralmeshgeneratorwithanexplicitFEformulation.Thesecodeswereusedinaverificationandvalidationstudiesforthe2014Mw5.1LaHabraearthquakeonTitan,totesttheaccuracyofthecodes,andtoexaminehowclosethepredictedgroundmotionsaretoobservations.Wemodelgroundmotionvariabilityforlargestrike-slipandblindthrustearthquakesincludingfrequenciesupto8Hz.Theearthquakesourceisobtained
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(a) CCA00 Vs 2.0 km−122˚ −120˚ −118˚ −116˚
34˚
36˚
38˚
2.42.62.83.03.2
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
(b) CCA06 Vs 2.0 km−122˚ −120˚ −118˚ −116˚
2.42.62.83.03.2
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(c) dVs 2.0 km−122˚ −120˚ −118˚ −116˚
34˚
36˚
38˚
−20−16−12−8−4
048
121620
%
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(d) CCA00 Vs 10.0 km
34˚
36˚
38˚
3.0
3.2
3.4
3.6
3.8
4.0
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
(e) CCA06 Vs 10.0 km−122˚ −120˚ −118˚ −116˚
3.0
3.2
3.4
3.6
3.8
4.0
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(f) dVs 10.0 km−122˚ −120˚ −118˚ −116˚
34˚
36˚
38˚
−15−12−9−6−3
0369
1215
%
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(g) CCA00 Vs 20.0 km
34˚
36˚
38˚
3.6
3.8
4.0
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
(h) CCA06 Vs 20.0 km−122˚ −120˚ −118˚ −116˚
3.6
3.8
4.0
−122˚
−122˚
−120˚
−120˚
−118˚
−118˚
−116˚
−116˚
34˚
36˚
38˚
(i) dVs 20.0 km−122˚ −120˚ −118˚ −116˚
34˚
36˚
38˚
−10−8−6−4−2
02468
10%
throughdynamicrupturepropagationusingSORDalongseveralrealizationsofroughfaulttopographies.Anensembleofsourcesismodeledbyvaryingthehypocenterlocationthatresultsinsimilarmomentmagnitudes.Theslip-ratedatafromtherupturemodelingisconvertedtoakinematicsourceandinputintothewavepropagationcodeAWP-ODC,whichincorporatesfrequency-dependentattenuation(Q(f),Withersetal.,2015)aswellasDrucker-Pragerplasticity.Wealsoincludesmall-scalemediumcomplexityinbotha1D-layeredmodelanda3DmediumextractedfromSCECCVM-S4includingasurfacegeotechnicallayer.ThemediangroundmotionfollowsasimilardecayascomparedtoGMPEswhenusingaQ(f)power-lawexponentintherange0.6-0.8.Nonlineareffectsareneededtoreducenear-fieldgroundmotiontoobservablelevelsinregionsofnear-surfacelowvelocitylayers.Small-scalemediacomplexityisobservedtodecreasethepolarizationratiotothatofsimilartoobservations.Theintra-eventvariabilityforthelayeredmodelsimulationsisnearobservedvaluesofsingle-stationstandarddeviation.Small-scaleheterogeneitycansignificantlyaffecttheintra-eventvariabilityatfrequenciesgreaterthan~1Hz,becomingincreasinglyimportantatlargerdistancesfromthesource.Theintra-eventvariabilityofoursimulationsintheCVMistypicallylargerthanthatfortheobservationsatfrequencies>1Hz.However,thisdiscrepancytendstodecreasewhensmall-scaleheterogeneitiesareincluded,suggestingtheneedforahighlycomplexvelocitymodeltofitgroundmotionvariability.Plasticeffectsinthemediumalsoreducethevariability,particularlyatdistancesclosetothesource.Weuseanon-associatedDrucker-PragernonlinearrheologyfollowingthereturnmapalgorithmintheAWP-ODCcodetomodelgroundmotionsfromtheM7.8ShakeOutscenariosourcedescription(M3)includingpermanentrockdeformation.Toaccountfornonlinearityinthefaultdamagezoneandinnear-surfacesediments,weusedatwo-stepmethod.Inthefirststep,thedynamicruptureprocessincludingplasticitywassimulatedinasmallcomputationaldomainusingtheCPUversionoftheAWP-ODCfinitedifferencecode.ThesedynamicsimulationswereperformedonNCSABlueWatersorTACCStampede,whereakinematicsourcewasalsogeneratedfromthedynamicsolution.Thesourcepartitions,whichrequireupto4Tbofdiskspace,werethentransferredtoOLCFTitanusingGlobusGridFTP.ThewavepropagationresultingfromthekinematicsourcewassimulatedonTitanwithAWP-ODC,usingalargervelocitymeshthatincludesmajorurbanareasintheregion.Thesesimulationsareperformedusing700GPUsduring4hoursforamaximumfrequencyof2Hzor5,600GPUsduring7hoursforamaximumfrequencyof2Hz.ResultsconfirmthatnonlineardeformationwouldoccurinawideareaaroundthefaultandintheSanBernardinoandLosAngelesbasins,reducinglong-periodgroundmotionsintheLosAngelesbasinbyupto50%withrespecttothelinearsolution(Figure3).Theimportanceofnonlinearityincreasesassimulationsareperformedathigherfrequencies.Weexploretheeffectsoffaultzonenonlinearityonpeakgroundvelocities(PGVs)bysimulatingasuiteofsurfacerupturingearthquakesinavisco-plasticmedium.Oursimulations,covermagnitudesfrom6.5to8.0,withseveral
realizationsofthestochasticstressdropforagivenmagnitude.Wetestthreedifferentmodelsofrockstrength,withfrictionanglesandcohesionsbasedoncriteriawhicharefrequentlyappliedtofracturedrockmassesincivilengineeringandmining.Weuseaminimumshear-wavevelocityof500m/sandamaximumfrequencyof1Hz.Inrupturescenarioswithaveragestressdrop(~3.5MPa),plasticyieldingreducesnear-faultPGVsby15to30%inpre-fractured,low-strengthrock,butlessthan1%inmassive,highqualityrock.Thesereductionsarealmostinsensitivetothescenarioearthquakemagnitude.Inthecaseofhighstressdrop(~7MPa),however,plasticityreducesnear-faultPGVsby38to45%inrocksoflowstrengthandby5to15%inrocksofhighstrength.Becauseplasticityreducesslipratesandstaticslipnearthesurface,theseeffectscanpartiallybecapturedbydefiningashallowvelocity-strengtheninglayer.Thesesimulationresultssuggestthatnonlineareffectsmayberelevantevenatlongperiods,especiallyforearthquakeswithhighstressdrop.
ProjectProductivity• ListPapersfromworkatALCFandincludeasentenceabouthowthisisrelated
totheproject.Examplesare:o “Thisworkwasdonewithaone-monthcampaignon32racksofMira.”o “TheseresultsweredeterminedusingdatageneratedfromanALCC
project.”o “TheseresultsarefromcollaboratorswhodidnothaveALCCtimebut
usedtheresultsofourwork.”o Etc.
• ListPresentationsworkatALCFhttp://www.alcf.anl.gov/projects/high-frequency-ground-motion-simulation-seismic-hazard-analysis
• Listanyotherawards,honors,mediacoverage,etc
Papers
Lee,E.-J.,P.Chen,T.H.Jordan,P.J.Maechling,M.A.M.Denolle,andG.C.Beroza(2014),Full-3-DtomographyforcrustalstructureinSouthernCaliforniabasedonthescattering-integralandtheadjoint-wavefieldmethods,J.Geophys.Res.SolidEarth,119(8),6421–6451,doi:10.1002/2014JB011346.Thispapersummarizesthefull3DtomographyresultsfromcomputationsattheArgonneLeadershipComputingFacility.
Xu,Z.,P.Chen,andY.Chen(2013),SensitivityKernelfortheWeightedNormoftheFrequency-DependentPhaseCorrelation,PureAppl.Geophys.,170(3),353–371,doi:10.1007/s00024-012-0507-3.
Thesis
Donovan,J.(2015),ForecastingDirectivityinLargeEarthquakesinTermsoftheConditionalHypocenterDistribution,PhDThesis,UniversityofSouthernCalifornia,154pp.
Presentations
SeismologicalSocietyofAmerica,AnnualMeeting2015(SSAAM)-Lee,E.,Thomas,H.J.,Chen,P.,Maechling,J.P.,Boué,P.,Denolle,M.,Beroza,G.,&Eymold,W.K.(2015)Full-3DTomographyofCrustalStructureinCentralCalifornia.Abstractandpresentation,2015SSAAnnualMeeting.BlueWaters2015Symposium-SCECpresentedtheresultsattheannualBlueWaterssymposium,includetheCyberShakecalculation,anexampleofaSCEC,NSFBlueWaters,andINCITEresearchcollaborativeeffort.LinkstothepresentationarepostedonaSCECwikiat:
http://scec.usc.edu/scecpedia/Blue_Waters_Symposium_2015NSFSoftwareInfrastructureforSustainedInnovation2015PIMeeting:-SCECmemberspresentedsoftwaredescriptionsandresearchresultsrelatedtoourFull3Dtomography(F3DT)andUnifiedCommunityVelocityModel(UCVM)workusingINCITEresourceataFeb2015NSFSoftwareInfrastructureforSustainedInnovation(SI2)meetingJan2015.TheNSFSI2programcurrentlyprovidesresearchfundingforsoftwareinfrastructureincludingF3DT,UCVM,AWP-ODC,andHerculesusedonourSCECINCITEresearchactivities.Moredetailsareavailableviathefollowinglinktothemeetingwebsite:https://share.renci.org/SI2PI2015/Lists/SI2PI2015Posters/View_01.aspx
NSCICoordinationMeeting(RestonVirginia)-SCECDirectoryThomasH.Jordanparticipatedinaninvitation-onlymeetinginOct2015concerningtheNationalStrategicComputingInitiative(NCSI)wherehediscussedSCECearthquakesystemscienceresearchbeingconductedusingALCF,OLCF,andNSFcomputingresources.
ExascaleComputingRFI:-SCEC’scomputationalscienceresearchgroupdevelopedadescriptionofourexascalecomputingneedsandcontributedittoaNSF/NIHHPCcommunity.SCEC’sdescriptionofourexascalecomputingneedsispostedhere:http://hypocenter.usc.edu/research/Exascale/SCEC_NCSI_Exascale.pdf
Supercomputing2015(SC15):SCECDirectoryThomasH.Jordanwasfeaturedasthe“InvitedTalkSpotlight”withhispresentationonSocietalImpactofEarthquakeSimulationsatExtremeScale.http://sc15blog.blogspot.com/2015/10/sc15-invited-talk-spotlight-societal.html
CodeDescription Pleasedescribeanymajordevelopmentorperformanceimprovementsthattookplacetoyourcode.Ifthedescriptionofthecodeissubstantiallydifferentfromearlierintheproject,pleasedescribe.SCECHPCcodeshavebeenfurtheroptimizedonINCITEresourcesin2015.Amajoreffortwastoportandintegratethenewnumericalalgorithmkernelplasticityandfrequency-dependentattenuationfeaturesintotheGPU-basedversionofAWP-ODC(OLCF).Simulationofnonlinearmaterialbehaviorrequiresanumberofadditionalvariables compared to linear computations. The added stencil kernel plasticityresultsincreasedghostcellregionfrom4to8layersinAWP-ODC,and17additionalvariablesaddedtotheoriginalcode.TheGPUcodeisrevisited,incollaborationwithDr. Peng Wang of NVIDIA. The combined new code moves advanced numericalalgorithms to a new level producing realistic seismograms at high frequencies,
whichhasbeentestedandvalidatedfromsciencerunswiththeversioncalledAWP-ODC-PLA-QF-v2.1.0.5.ThiseffortissupportedbyNCSABlueWatersPAIDprogram.Theplasticitykernelwasabletoachieveeffectivememorythroughputof223GB/safter the tuning, or 89% of the peak, the tuned code has seen 13% overallimprovement. In addition, we have implemented and validated CUDA multi-streamingonKeplerforexecutingmultiplekernelsconcurrentlyonOLCFTitanandNCSA Blue Waters. Multistreaming was available on Fermi with 16-wayconcurrency,butCUDAstreamsmultiplexintoasinglequeue.OnKeplerthe32-wayconcurrency is allowed at full stream level, with one work queue per stream,without inter-stream dependencies. The new implementation of multi-streamingcodingisvalidatedandresults inaspeedupof17%intermsofwallclocktimeonKeplerXK7.Weplantoapplythismulti-streamingfeaturetothebaselineAWP-ODCandalsothenewplasticitycodetotakethefullperformanceadvantage.The near future plan is to complete the optimization for the velocity kernel insimilar way like performed for plasticity and stress kernel, also add parallel filewriting capability to thenewcode.Weplan to startmediumsize science runsonINCITEresources later inSpring2016,after thenewAWP-ODC implementation iscompleteandvalidated.SCECmakesextensiveuseofscientificworkflowstorunourCyberShakemodelcalculation.CyberShakeproductionrunsareourlargestheterogeneousensemblesimulationsthatdependheavilyonworkflowtechnology.ThescaleandcomplexityoftheCyberShakecalculationrequirestheautomationandreproducibilityprovidedbyworkflows.WewouldliketorunourworkflowsonbothMiraandTitan.SCEC’sexistingCyberShakeworkflowimplementationisbasedonaNSF-developedsoftwarestackthatincludesCondor,CondorDAGManager,andPegasus-WMS.Inthepast,SCEChasnotbeenabletorunourworkflowsonINCITEresourcesprimarilybecauseourworkflowsolutionrequiresremotejobsubmission.Inourcurrentworkflow-computingmodel,anexternalcomputer,atSCEC,runsaCondorDAGManagerjobqueue.Jobsaresubmittedtotheresourceprovider’sjobqueueatareasonablerate.HPCproviders,includingTACCStampede,andNCSABlueWatershavesupportedthisworkflow-computingmodelwithGlobusGRAM-basedjobsubmission.Thisyear,SCECworkedwithUSCPegasus-WMSworkflowtooldevelopers,andTitan’stechnicalgrouptodevelopasolutionthatwillenableSCECtorunourworkflowsonTitan.TitanprovidesthelargeGPUcountneededforthecomputeintensivepartofourCyberShakecalculations.Weareworkingtodeveloptechnicalsolutionthatwillenableustoliketorunthehigh-throughputpartoftheworkflowonTitan.OnceworkingonTitan,wewillworktoimplementthissolutiononMiraaswell.
1. WhatisthecontributionmadebyLCFstaffand/orprograms(liaison/catalyst
collaboration,vendorsupport,userassistance,otherstaff,training,etc.)SCECinteractedwithALCFCatalystsandOLCFLiaison’sduring2015onstart-upcalls,onprogressreports,andintechnicaldiscussions.SCEC’sALCFCatalysthelpedusmigratelargedatasetsatALCFbetweenINCITEprojectswithoutinterruptingongoingactiveresearchusingthedata.SCEC’sOLCFliaisonshelpedcoordinatetechnicaldiscussionbetweenourworkflowdevelopmentgroupandtechnicalgroupsinsupportofourworkflowdevelopment.
NextSteps• Pleaselisttheexpectednextstepsforthiswork.Willyoubeworkingon
analysis?Ifso,where?Doyouneedadditionalallocations?Arethereupcomingpapersorpresentationsofthematerial?
• Aretherechangestocomingyearworkifyourprojectiscontinuing?Weareontracktocontinueourprogressforyear2milestones.Ourfull3DtomographyworkatALCFwillbemodifiedtorunathigherfrequencies.Thiswillrequirelargerparalleljobs.WewillworkwithALCFstaffsothatasweincreasethefrequencyofourtomographyresearch,ourroutinetomographycalculationswillscaleuptouse8racks,ormore,ofMira.
OtherCommentsPleaseanswerasapplicable:Hasthesupportreceivedfromthefollowingbeenbeneficialtoyourprojectteam?Citeexamplesifpossible• UserAssistanceCenter• CatalystsorPerformanceEngineering• VisualizationandAnalysisTeam• Operations
AnyadditionalfeedbackfromyourprojectteamfortheALCF?Pleaseincludeanyothercommentsorrequestsyouhaveaboutthefacility,allocationprogram,etc.
Recommended