Smarter Balanced Reporting Runbook, Deployment ... Smarter Balanced Reporting Runbook, Deployment,

  • View
    226

  • Download
    0

Embed Size (px)

Text of Smarter Balanced Reporting Runbook, Deployment ... Smarter Balanced Reporting Runbook, Deployment,

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    SmarterBalancedReportingRunbook,Deployment,Administration

    AWSVersion

    Preparedfor:

    by:

    Runbook,Deployment,Administration Page1of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    RevisionHistory

    RevisionDescription Author/Modifier Date

    InitialRelease(DRAFT) AnnaGrebneva(Amplify) 2014.07.09

    AddedTestScoreBatcher DorisIp(Amplify) 2014.11.03

    AddedAWS ClintonWolfe 2015.05.08

    AddedDependencies AnnaGrebneva 2015.07.15

    Runbook,Deployment,Administration Page2of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    TableofContents

    1Summary2DataCapacityPlanningAssumptions3HardwareMinimumRequirements4RequiredInfrastructureComponents

    AccessControlandSecretsManagementMonitoringandLoggingAutomationNetworkConfiguration

    SubnetConfigurationSecurityGroupConfiguration

    OtherNeeds5SoftwareRequirements5.1ServiceDependencies

    ServiceShutdown/StartupOrder5PlanningforAvailability6Archival,BackupandRecovery7Installation&Configurations

    7.1ReportingWebServers7.1.1Installation7.1.2ApacheConfiguration

    7.1.2.1MPMMode7.1.2.2ConfigurationFilestoAdd7.1.2.3ConfigurationFilestoRemove

    7.1.3SmarterApplication7.1.3.1Generatingsmarter.ini7.1.3.2ConfigurationFileforAuthenticationbetweenOpenAMSAMLandSmarter7.1.3.3AddingCustomMetadataConfiguration7.1.3.4Configuringsyslog7.1.3.5CreatingaDatabaseSchemaforProductionDatabase

    7.2LoadBalancer7.3.1Installation7.3.2Configuration

    7.4PDFWorker[Ansiblename:reportingworkerpdf]7.4.1Installation7.3.2Configuration

    7.3.2.1GlusterFS7.3.2.3celerydservices

    7.4PDFPreGenerator[Ansiblename:reportinggeneratorpdf]

    Runbook,Deployment,Administration Page3of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    7.4.1Installation7.4.2Configuration

    7.4.2.1smarter7.4.2.2WSGI

    7.5ExtractMessenger[Ansiblename:reportingrabbitextract]7.5.1Installation7.5.2Configuration

    7.6ExtractWorker[Ansiblename:reportingworkerextract]7.6.1Installation7.6.2Configuration

    7.7Cache[Ansiblename:memcached]7.7.1Installation7.7.2Configuration

    7.8CacheWarmer[ansiblename:reportingcachewarmer]7.8.1Installation7.8.2Configuration

    WSGIConfigurationforSmarterapplicationspecificWSGIconfiguration

    smarterGeneratingSmarter.iniConfigurationsspecifictoCacheWarmerinSmarter.iniConfigurationforDemographicsFilters

    7.9DatabaseMaster[ansiblename:dbmaster]7.9.1Installation

    postgresConfiguringpostgresmaster

    7.10DatabaseReplica7.10.1Installation

    postrgrescelerydedmigrate

    7.11DatabaseLoadBalancer7.11.1Installation

    7.12DatabasePool[ansiblename:reportingdbpgpool]7.12.1Installation

    pgpoolforSmarterBalancedReportingwebfrontendpgpoolforExtracts

    7.13Smoketestforsmarterfunctioning7.14LandingZone

    7.14.1Installation7.14.2Configuration

    edsftpConfiguringchrootGeneratingSmarter.inifile

    Runbook,Deployment,Administration Page4of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    CreatingGroupsforSFTPusersCreatingTenantAccountsStartingEdSFTPwatcherservice

    7.15Loader7.15.1Installation7.15.2Configuration

    celerydudl2Generateudl2_conf.iniGeneratesmarter.iniInitializetheUDL2databaseEnsureGPGkeysarecopiedto/opt/edware/keysMountWorkZonesdirectoriesfromGlusterEnsureOutgoingHTTPandHTTPsportsareopenedStartedudl2filegrabberandedudl2triggerservicetowatchforincomingfilesbeingcopiedtoworkzone

    7.16LoaderMessenger7.16.1Installation7.16.2Configuration

    7.17LoaderDatabase7.17.2Configuration

    postgresAllowusername/passwordbasedAuthenticationAllowclientconfigurationCreateDatabaseUserCreateUDLdatabaseCreateudl2UserandGroupPrepareWorkZoneDirectory

    7.18DatabaseStaging7.18.2ConfigurationAllowclientconfigurationCreateDatabaseUserCreatetheedwaredatabase7.19.1Installation

    7.20Migrator7.20.1Installation7.20.2Configuration

    RabbitMQedmigrate

    Settingupthe.inifile7.21HTTPSPickupZoneWebServer

    7.21.1Installation7.21.2hpzwebConfiguration

    7.21.2.1Configureapache

    Runbook,Deployment,Administration Page5of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    7.21.2.2glusterfs7.21.2.3encfs7.21.2.4ApplicationConfiguration

    7.22.2hpzdbConfiguration7.22.3Heartbeat

    7.23ScoreBatcherWebServer7.23.1Installation

    7.22.2Configuration7.23.2.1ApacheConfiguration7.23.2.2ApplicationConfiguration

    7.24ScoreBatcherMessenger7.24.1Installation7.24.2Configuration

    7.25ScoreBatcherWorker7.25.1Installation7.25.2Configuration

    CustomXMLMetadataDirectoryStructure7.26ScoreBatcherTrigger

    7.26.1Installation7.26.2Configuration

    7.26ScoreBatcherDatabaseServer7.26.1Installation

    7.26.2InstallationCreatingaDatabaseSchema7.26.3Configuration

    8StartingApplications9Logging&Monitoring

    9.1WebServer9.2HTTPPickupServer9.3PDFMessenger[Ansiblename:reportingrabbitservices]9.4PDFWorker[Ansiblename:reportingworkerpdf]9.5PDFPreGenerator[Ansiblename:reportinggeneratorpdf]9.6ExtractMessenger[Ansiblename:reportingrabbitextract]9.7ExtractWorker[Ansiblename:reportingworkerextract]9.8Cache[Ansiblename:memcached]9.9CacheWarmer[ansiblename:reportingcachewarmer]9.10DatabaseMaster9.11DatabaseReplica9.12DatabaseLoadBalancer9.13DatabasePool9.14LandingZone9.15Loader9.16LoaderMessenger9.17LoaderDatabase

    Runbook,Deployment,Administration Page6of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    9.18DatabaseStaging9.19Migrator9.20ScoreBatcherWebServer9.21ScoreBatcherMessenger9.22ScoreBatcherWorker

    10Troubleshooting11Maintenance

    11.1Cache11.2Database11.3HTTPPickupZone

    AppendixA:ApacheConfigurationFilesconf/httpd.confconf.d/edware_worker_mpm.confconf.d/rewrite_reporting_https.confconf.d/rewrite_reporting_slash.confconf.d/rewrite_tsb.confconf.d/ssl_main_server.confconf.d/ssl_on.confconf.d/wsgi_edware.confconf.d/wsgi_frs.confconf.d/wsgi_swi.confconf.d/wsgi_tsb.confconf.d/xsendfile.conf

    AppendixB:ServiceProviderMetadata

    Runbook,Deployment,Administration Page7of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    1 Summary ThisdocumentisforSystemAdministratorswhowillbeoperatingtheSystem.Itcontainsinstructionsonhowtoinstall,scale,andmaintaintheSmarterBalancedDataWarehouseandReportinginstance.

    2 Data Capacity Planning Assumptions 1. SmarterBalancedReportingDatabaseStorage:

    a. StudentRegistration:0.5kbperstudent.b. StudentAssessments:2kbperassessmentand0.5kbperstudent.c. Allocatesimilarsizetothewholepostgresqldatabaseforpostgresqlpointintime

    recovery.d. Keepdiskusagelessthan50%tohelpDatabaseAdministratorstoperform

    postgresvacuumoperationsefficiently.2. PDFfileStorage:

    TheSmarterwebapplicationwillpregenerategrayscalePDFs,andcolorPDFswillbegeneratedwhenauserrequests.Therefore,itisgoodpracticetoestimatediskspacefor100%diskstorageforgrayscalePDFsfilesand50%diskstorageforcolorPDFs.

    a. color85kb/assessmentoutcome.b. grayscale75kb/assessmentoutcome.

    3. HistoricalAssessmentArchive:a. 50mbassessmentcsvfor100kperassessment.

    4. IndividualItemResponse(a.k.a.ItemLevel)DataStorage:a. Averagesizeoffileaftercompressionis200kb.b. Astudentgeneratesaround5itemlevelfilesperschoolyear.c. SBACrequiresstoringof10yearsofstudentitemlevelxmlrawdata.

    5. HTTPPickupZoneStorage:a. [PLACEHOLDER:performanceandloadtestingscheduledaspartofM6].b. Databaseforpickupzoneoperation.

    6. LandingZoneStorage:a. 0.5kbperstudentperassessment.

    7. LoaderDatabaseStorage:a. [PLACEHOLDER:performanceandloadtestingscheduledaspartofM6].

    3 Hardware Minimum Requirements Thenumberofinstancesrequiredforeachmachinetypeiscorrelatedtothreeindependentcriterias:thenumberoftenants/statesinthesystem,thenumberofusersaccessingthesystem,andthenumberofstudentsinthetenant/state.

    Runbook,Deployment,Administration Page8of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    Table1Sampleconfigurationfordatawarehouseandreportingfor1tenantcontaining2Mstudentsand100Kusers.

    Application AvgUsage(3%userconcurrency)instances

    PeakUsage(10%userconcurrency)instances

    Cores(vCPUperinstance)

    Memory(GBperinstance)

    AWSinstancetype

    nonOSportionStorage(GB)

    reportingweb(mainwebapplicationserver)2

    9 30 4 8 m3.large

    loadbalancers2

    2 2 2 2 ELB

    dbmaster1 1 1 8 64 r3.2xlarge 1000

    reportingdbslave(replicasforwebfrontend)2

    2 7 8 64 r3.2xlarge 1000

    extractsdbslave(replicasforextraction)2

    1 3 8 64 r3.2xlarge 1000

    reportingdbpool2

    2 3 4 8 m3.xlarge

    memcache3 2 2 8 64 m3.2xlarge

    reportingrabbitservices2

    2 2 4 8 m3.xlarge

    reportingworkerpdf3

    2 2 8 64 r3.2xlarge

    reportinggeneratorpdf1

    1 1 2 8 m3.large

    reportingrabbitextract2

    2 2 4 8 m3.xlarge

    reportingworkerextract2

    2 7 8 32 m3.2xlarge

    reportingcachewarmer1

    1 1 2 8 m3.xlarge

    Runbook,Deployment,Administration Page9of96 SmarterBalancedReporting(RFP15)

  • SmarterBalancedReportingRunbook,Deployment,AdministrationAWSVersion

    lz(landingzone)1

    2 2 4 8 m3.xlarge 50

    hpzweb(httpspickupzonewebapp)5

    3 9 4 8 m3.xlarge

    hpzdb5 1 1 4 8 m3.2xlarge 10

    udl(universaldataloader)3

    2 2 4 8 m3.2xlarge

    udlrabbit1 2 2 4 8 m3.xlarge

    udldb1 1 1 8 32 m3.2xlarge 1000

    migraterabb