AwardNo.1541215 4/1/2016NSF-Arlington,VA

AwardNo.1541215

BuildingaFederatedCloudModel:AristotleCloudFederation

DavidLifka,PIDirector,CornellUniversityCenterforAdvancedComputing(CAC)InterimCIO&VicePresident,CornellInformationTechnologies

lifka@cornell.edu

ThomasFurlani,Co-PIDirector,CenterforComputationalResearch(CCR),UniversityatBuffalo

furlani@buffalo.edu

RichWolski,Co-PIProfessor,ComputerScience,UniversityofCalifornia,SantaBarbara

Co-Founder,EucalyptusSystemsrich@cs.ucsb.edu

AwardNo.1541215

AristotleOverview&Goals• $5million,5-yearNSFawardtodeployafederatedDIBBs

cloudatCornell,U.Buffalo,UCSantaBarbara– 7scienceteams,over40globalcollaborators

• Requiringflexibleworkflows/analysistoolsforlarge-scaledata• Representingdiversityofanalysisrequirementsandcloudusagemodalities

– Earthandatmosphericsciences,finance,chemistry,astronomy,civilengineering,genomics,foodscience

• Projectgoals– Optimize“timetoscience”– Demonstratethevalueofsharingresourcesanddataacross

institutionalboundaries

3

AwardNo.1541215

34

5

6

1

2

7

Multi-DisciplinaryCollaborationsinaMulti-CampusCloudFederation

AwardNo.1541215

ProjectWorkBreakdownStructure

5

CloudFederation Project Management,Oversight&Reporting

CU0.10FTEUB0.01,UCSB0.01TotalFTE:0.12

DIBBsAcquisition,Installation,Configuration,Testing&Maintenance

CU 0.08,UB0.25,UCSB0.1TotalFTE:0.43

CloudFederation PortalCU 0.33,UB1.0,UCSB0.66

TotalFTE:1.99

PortalDevelopment&MaintenanceCU 0.28,UB0.25,UCSB0.20

TotalFTE:0.73

Allocations&AccountingIntegrationCU 0.02,UB0.1,UCSB0.1

TotalFTE:0.22

OpenXDMoDIntegrationCU 0.02,UB0.65,UCSB0.0

TotalFTE:0.67

Eucalyptus&QBETSIntegrationCU 0.01,UB0.0,UCSB0.36

TotalFTE:0.37

ResearchTeamSupportCU 0.30,UB0.75,UCSB0.31

TotalFTE:1.36

AwardNo.1541215

ProjectGovernance

6

DirectorNSFProgramManager

ExecutiveSteeringCommittee

ScienceTeamAdvisoryCommittee(STAC)

ExternalAdvisoryCommittee(EAC)

UserCommunity

AwardNo.1541215

ExternalAdvisoryCommitteeIntroductions

7

Name Affiliation Project Email

Amy Walton NSF Aristotle awalton@nsf.gov

Adam Brazier CAC Aristotle brazier@cornell.edu

Susan Mehringer CAC Aristotle susan@cac.cornell.edu

Paul Redfern CAC Aristotle red@cac.cornell.edu

Resa Reynolds CAC Aristotle rda1@cornell.edu

Jamie Kinney AWS SciCo jkinney@amazon.com

Dmitrii Zagorodnov HPE Eucalyptus cedzc@hpe.com

Ben Rosen Dell Big Data & Cloud ben_rosen@dell.com

Craig Stewart IU Jetstream stewart@indiana.edu

John Towns NCSA XSEDE jtowns@illinois.edu

Rick Wagner SDSC Comet rpwagner@sdsc.edu

Ian Foster UC Globus foster@cs.uchicago.edu

Steve Johnson WCM NIH CTSC johnsos@med.cornell.edu

AwardNo.1541215

ProjectOfficeUpdate(Q1/Q2)• CompletedCooperativeAgreement&PEP(12/18/15)

– EstablishedExternalAdvisoryCommittee• CompletedUBandUCSBsubcontracts• Establishedbiweeklyteamcalls• Establishedmonthly/quarterlyreportingprocess• Discussionsaroundenablingtechnologies

– OpenXDMoDdatacollectionforAristotlefederation@SC15– CycleComputingleverage@SC15– SupercloudwithCornellComputerScienceandJetstream– QBETSintegrationintoOpenXDMoDandGlobusGenomics– Keyportalcomponentsandcapabilities– AWSSciCopartnership

• Agreementonallocationsandaccountingprinciples

8

AwardNo.1541215

InfrastructureUpdate(Q1/Q2)• CornellRedCloudinproduction for5years

– RedCloudEucalyptusupgradedto4.2– FederationcapabilitiestestedbetweenIthacaandWCMNYC– CurrentAristotleportalsiterunninginRedCloud– January2016firstinstallmentofDIBBinfrastructureordered/installationcomplete

• UBLakeEffectcloudinstalled/operational10/16/15– LakeEffectEucalyptusupgradedto4.2– IntegratingintoexistingUBcoreservices– LeveragingEucaAvailabilityZones– January2016firstinstallmentofDIBBsinfrastructureordered/installationnearlycomplete

• UCSBEngineeringComputing Infrastructurecloudinproduction for2years– UCSBEucalyptuscloudupgradedto4.2– WorkingonInCommonsetup– FirstinstallmentofDIBBsinfrastructureordered/arrivingnow

9

AwardNo.1541215

PortalUpdate(Q1/Q2)• Goal:InformedBursting

– AcrosstheAristotlefederation– NSFClouds– AWS

• Definedportalcomponentsanddeveloperresponsibilities• CompletedInCommonintegrationproofofconcept

– SupportfromHPEHelionEucalyptusteamforInCommonaccesstoEucaPortalindevelopment• CompletedsetupofAristotleRTsystematCornell

– help@federatedcloud.org– Establishedticketroutingroles

• CompletedAllocations&Accountingdatabaseschemaanddatabase– BasedonstreamlinedversionofCornellCACAllocations&Accountingsystem

• WilldrawbasicuserdocumentationfromRedCloud• Utilizationandallocationbalancereporting

– AbilitytoshowDIBBsusagevs.localinstitutionalusage• IntegrationofXDMoDandQBETs• GlobusOnlineupatCornell;UBandUCSBinprogress

10

AwardNo.1541215

ResearchTeamsUpdate(Q1/Q2)• 7initialscienceteams

– UseCase1:ACloud-BasedFrameworkforVisualization&AnalysisofBigGeospatialData(UB)– UseCase2:GlobalMarketEfficiencyImpact(UB)– UseCase3:HighFidelityModelingandAnalyticsforImprovedUnderstandingofClimate-

RelevantAerosolProperties(CU)– UseCase4:TransientDetectioninRadioAstronomySearchData(CU)– UseCase5:WaterResourceManagementUsingOpenMORDM(CU)– UseCase6:MappingTranscriptomeDatatoMetabolicModelsofGutMicrobiota(CU)– UseCase7:Multi-SourcedDataAnalyticstoImproveFoodProduction(UCSB)

• Scienceteammeetingsstartedfromday1– Understandcurrentworkflowsandprocesses– Documentsoftwareandhardwarerequirements– Developcloudadoptionstrategy– StartcreatingVMs(e.g.,geospatialanalytics,financeanalytics,Windowsinstancefor

MATLABVMsalreadycompleted)– Demoproofofconceptstoscienceteams

11

AwardNo.1541215

UseCase1:ACloud-BasedFrameworkforVisualizationandAnalysisofBigGeospatialData(UB- Chandola)

• What?– Integrate,visualizeandanalyzeheterogeneousgeospatialdatafrom

aroundtheworld• WhyAristotle?

– Offloadvariablecomputationaldemands fromlocalinfrastructure– Createanexpertservicehubforwidelydistributeddata– Scalableandelasticcomputing

12

AwardNo.1541215

UseCase2:GlobalMarketEfficiencyImpact(UB- Tin,Roesch,Wolfe)• What?

– Measuremarketefficiencieswithhighvolumesofhistoricalandcurrentintradaymarketdata

• WhyAristotle?– Bursttointegratenewdata– TensofTBsmovedtoavailableprocessingresources– Sharedataandpipelinesaroundtheworldforcollaboration

andreproducibility

13

AwardNo.1541215

UseCase3:High-FidelityModelingandAnalyticsforImprovedUnderstandingofClimate-RelevantAerosolProperties(CU- Pryor)

• What?– Modelthebehaviorofatmosphericaerosolparticlestopredictand

ameliorateclimateandhumanhealthimpacts• WhyAristotle?

– Bursttoprocessnewdata– Shareahigh-valueprocesseddatasetofgeneralinterest– Buildan extensibletoolkitforresearchers,transferrabletoother

groups

14

AwardNo.1541215

UseCase4:TransientDetectioninRadioAstronomySearchData(CU- Cordes)• What?

– Low-latency/on-demandsearchfortime-varying,includingone-time,signalsinradioastronomydata

– BuildonexistingCornell/CACcollaboration*• WhyAristotle?

– Burston-demandtoanalyzenewdata– High-speedtransportofverylargedatasets intoandaroundthe

federation

*"ArepeatingFastRadioBurst,”Nature531,202–205(10March2016)

15

AwardNo.1541215

UseCase5:WaterResourceManagementUsingOpenMORDM(CU- Reed)• What?

– Simulatecomplexwaterresourcemanagementoutcomeson-demandtoachievesubstantialbeneficialenvironmentalimpactonmunicipalitieslocatedthroughouttheUS

• WhyAristotle?– Startcoarselyparallelcomputationson-demand– Verylargecomputations, idealfortransfertoAWSorotherclouds– Makeavailabletomanymunicipalities

16

AwardNo.1541215

UseCase6:MappingTranscriptomeDatatoMetabolicModelsofGutMicrobiota(CU- Douglas)

• What?– Usehostandgut-microbialexpressiondatatoinferwhichmetabolic

reactionsoccurineachorganismasaresultofdietaryinputs• WhyAristotle?

– Heterogeneous instancetypesandsizeswithhighavailabilitytoallowforunpredictablecomputationaldemands

– AllowsgenerationofreproduciblecomputationalbiologypipelinesintheformofVMsorVMconfigurations

17

AwardNo.1541215

UseCase7:Multi-SourcedDataAnalyticstoImproveFoodProduction(UCSB/SedgwickReserve- McCurdy)

• What?– Interdisciplinaryanalysisofdatafromawidevarietyofsources

(includingthegeneralpublic)tostudyandimproveecologicaloutcomes

• WhyAristotle?– Scalableandportableinfrastructure– Accesstomultipledatasources,manyofwhicharealreadyin

publicandprivateclouds

18

AwardNo.1541215

OutreachActivities• CASCPresentation

LifkapresentedinfoonAristotleatthefallCoalitionforAcademicScientificComputation(CASC)meetinginArlington.Severalinstitutionsexpressedimmediateinterestinjoiningthefederation.Lifkaexplained thatiscertainlyagoalandwewillreportbacktoCASCwhenweareready.

• MediaOutlets/SocialMediaGeneratedmediacoverageacross23globalmediaoutletsincludingmajorIToutlets (CIO,CampusTechnology,GovernmentTechnologyNews,NetworkWorld)aswellasHPCverticals(ACMTechNews,HPCwire,InsideBIGDATA, ScientificComputingWorld).Astrongsocialmediapresence(Twitter,LinkedIn,etc.)wasgeneratedincollaborationwithNSF,AWS,andotherprojectpartners.

• CloudLaunchBackgrounderCornellproducedaCloudLaunchbackgrounderthatdescribesitsfeaturesandbenefits:https://www.cac.cornell.edu/technologies/CloudLaunch.pdf.

• SC15ConferenceTheAristotlefederationwasfeaturedatCornell’sandUB’sSC15exhibitsandnews andbackgroundersweredistributed toattendees.

19

AwardNo.1541215

OutreachActivities(cont.)• NationalMediaCoverage

ACMTechNews- CornellLeadsNewNSFFederatedCloudProjectCampusTechnology-CornelltoLeadNSF-FundedCloudFederationforBigDataAnalysisCloudwards-Aristotle:AcademicFocusedCloudFundedCIO- Universityresearchersget$5Mgranttobuild'AristotleCloud'CloudStrategyMagazine-CornellLeadsNewNSFFederatedCloudProjectDataCenterTalk-CornelltoHeadthe$5MFederalCloudComputingProgramGlobalWirelessResearch-Universityresearchersget$5Mgranttobuild'AristotleCloud'GovernmentTechnologyNews-WisdomoftheClouds:AristotleCloudFederationHPCwire-CornellLeadsNewNSFFederatedCloudProjectInsideBIGDATA-CornelltoLeadAristotleCloudFederationforResearchNetworkWorld-Universityresearchersget$5Mgranttobuild'AristotleCloud'NextGenerationCommunications-CornellLeadsNewNSFAristotleCloudFederationProjectScientificComputing-Cornelltolead$5MNSFFederatedCloudProjectScientificComputingWorld-Willthecloudchangescientificcomputing?WebHostIndustryReview-NationalScienceFoundationSponsors$5MillionCloudProject

• InternationalMediaCoverageCIOL(India)-CornellUniversitytodevelopfederatedcloudComputerWorld(Australia)-Universityresearchersget$5Mgranttobuild'AristotleCloud'PrimeurMagazine(UK)-CornellleadsnewNationalScienceFoundationfederatedcloudprojectTechWorld(Australia)-Universityresearchersget$5Mgranttobuild'AristotleCloud'

• LocalMediaCoverageCornellChronicle-Aristotle:AfederatedcloudforacademicresearchCornellDailySun-CornelltoLead$5MFederalCloudComputingProgramIthacaJournal-Cornelltoheadefforttostreamlinedataflows

20

AwardNo.1541215

SharingandWorkingwiththeCommunity• We’redevelopingandidentifyingcloud-relatedtools

– QBETShttp://www.cs.ucsb.edu/~nurmi/nurmi_qbets.pdf

– OpenXDMoDwithcloudsupport http://xdmod.sourceforge.net/

– CloudLaunch(potentialworkwithCycleComputingandAWS)https://www.cac.cornell.edu/technologies/CloudLaunch.pdf

– Supercloud(testingonJetstream)http://fireless.cs.cornell.edu/publications/supercloud.pdf

• Wewanttocreatea”Cloudy”communityfocusgroup– Identifyandsharecloudtools,services,andsolutions– Leveragenewcomputingandscalingparadigms(e.g.,containerization)– Helpeachotherandthegreaterresearchcommunityachievefastertimeto

science

21

AwardNo.1541215