Upload
simon-j-mudd
View
483
Download
3
Embed Size (px)
Citation preview
HowtosetupOrchestratortomanagethousandsofMySQLservers
SimonJMudd |3rd October2017
SessionSummary
• Whatisorchestratorandwhyuseit?• Whathappensasyoumonitormoreservers?• Featuresaddedtomakeitscaleandimproveusability• Usingorchestratoratsmallerscale• Wayforward
1
Booking.com
• Oneofthelargesttravele-commercesitesintheworld• part ofthePricelineGroup(NASDAQ:PCLN)• Weofferaccommodationin228countries• Ourwebsiteandcustomerservicein40languages• Morethan15,000employeesin204officesin70countries
• WeusethousandsofMySQLservers:• weuseorchestratortomanagethetopologyandhandlemasterandintermediatemasterfailures
2
WhatisOrchestratorandwhyuseit?
3
Orchestrator
4
Orchestrator
• WrittenbyShlomi Noach• hestartedonthisatoutbrain andisnowworkingatgithub.com• Heintroducedbooking.com toorchestratorabout3yearsagowhenwewerelookingforsomethingtohandlefailoversautomatically
5
Orchestrator
• PeriodicallymonitorsMySQLserversandcheckstheirhealth• Handlesmasterfailover,butalsodoesmuchmore…• GUItomanageandvisualise topology– veryhandy• CLItodothesametasks– usedforscripting• APIcallstorunatadistance• NeedsaDBbackendtostorestate.
• NormallyMySQLbutcanbeSQLite
• Writteningo
6
Orchestrator
Whatfailuresdoesithandle?• Masterfailures
• Optionalhookstoexternalsystemswhichneedtobeawareofthesefailures
• Intermediatemasterfailures• Doesnot careaboutleafslavesorapplications• WorkswithOracleorMariaDB GTID• WorkswithoutGTID:CanaddPseudo-GTID (eventsinjectedonthemasterareusedtofindamatch)sononeed tomigrate
• Handlesmulti-leveltopologies7
OrchestratorGUI
8
OrchestratorGUI
9
OrchestratorGUI
10
Orchestrator
TopologyManagement• DraganddropusingtheGUI
• Moveoneslaveabout• Moveallslaves
• ScriptablerelocationfromthecommandlineorusingAPIcalls
11
Whathappensasyoumonitormoreservers?
12
Whathappensasyoumonitormoreservers?
• Integrationneededwithinternalinfrastructure• Deployment:tellorchestratortodiscoverandforgetservers*• Determinecandidatemasters• Handlespecialcases:
• testMySQLversions,specialsetups(black- orwhite-listserversorclusters)
• MakeorchestratorHA• Monitororchestratorbehaviour andperformance• Providewideraccesstodifferenttypesofuser
13*Itcanautomaticallydetectnewserversinanexistingclusterbutnotnewdetectnewclusterswithouthelp
IntegrationwithInternalInfrastructure
• Populatethemetadatadb onthemasterto:• Maphostorinstancenamestomorefamiliarclusternames• Howtodeterminereplicationdelay• Configurationofacceptablelevelsofreplicationdelay
• Addandremovalofservers/instancesastheyaredeployedorremovedfromservice
• SetupofPseudo-GTID(ifnotusingGTID)
14
IntegrationwithInternalInfrastructure
• Addfailoverhooksformonitoring,notificationandtotakesite-specificactions(tellothersystemsaboutthenewmaster)
• Selectionofcandidatemasters• Blacklistingserverswhicharenotsuitable:backupservers,testservers,serversinthewrongnetworkareas…
15
BetterVisibility
• Improveorchestratordeploymentvisibility• Foreachrunningapp:showhost,version,uptime• Showtheactivenodeandhowlongit’sbeenactive
• AuditingofMySQLfailuresandrecoveryviatheGUIisgoodandimproving
• noneedtosearchthelogs
16
BetterVisibility
17
Featuresaddedtoscaleandimproveusability
18
Performance
Wefoundbottlenecksespeciallyonstartup• Trytodiscoverseveralthousandmysql serversatonceandupdatethebackendatthesametimeàmax_connections exceeded
• Multiplegoroutinestryingtopollthesamestuckserver
Solution:• FIFODiscoveryqueuewhichavoidsduplicatesandlimitsmaximumdiscoveryconcurrency
19
Performance
Howtofigureoutwhat’sgoingon?• Understandingloggingishardatthisscale– toomuchnoise• Nodiscoverymetricstoseeproblemsatserveroraggregatelevel
Solution:• CollectdiscoverymetricsandkeepforNseconds• Logdiscoverytimesindebugmode• Provideinterfacetoretrieveraworaggregatevaluestouseinmonitoringsystems
20
Performance
Discovery(Poll)times
21
Performance
Discovery(Poll)counters
22
Performance
• Aclientupgrademightupgradethedatabasewhichotherolderappswerestillusing
Solution:• Makeauto-upgradeofthedatabaseoptionalsotheDBAcontrolsthis
23
Performance
• Crosszone(dc)accesschangesperformanceprofilesignificantlyandcausedproblems
• orchestratorappsaresupposedtobeeasytoreplaceandlocationshouldnotmatter
• latencycanbearealenemySolution:• Batchupdatesofsomedataintosmallernumberoflargerinserts• Collectmetricsonthesetimings• Catchdiscoverieswhichtaketoolong(internalcodebottlenecks)• Visibilityofthemetricsmadeiteasiertolocatecauses
24
Performance
• Specialconnectionssettings• "MySQLOrchestratorMaxPoolConnections":controlgopoolsize• "MySQLConnectTimeoutSeconds":1
• don’twastetimewaitingtoconnecttoadeadserver
25
Performance
golang specific-isms• Orchestratorbydefaultusesdatabase/sql andbydefaultsendsaquerywithparametersusingMySQL’sPrepare/Executesyntax
• Thisgenerates2rtt’s andonslowerconnectionscanaffecttheelapsedtimetocompleteaquery
• OptionstodisablethisbyinterpolatingparametervaluespriortosendingSQL
• Go(orchestratorcode)isquitehappytotry topoll10,000serversatonce
• Sometimesthatisnotsensible• Throttlingtoavoidthunderingherdisnecessary
26
OrchestratorHA
• Morethenoneorchestratorserverperzone/dc• Someupgradesreallyeasy– justrestartwithnewbinaries
• Commonendpointvialoadbalancer• Simplerforusers• worksforapi callsandmaysimplifyfirewallrules
27
28
OrchestratorHA
LoadBalancer
app1 app2 app3 app4
nginx1 nginx2 nginx3 nginx4
backend
Zone1 Zone2
OrchestratorHA
MightIhavemorethanoneorchestratorcluster?• Yesforactivedevelopment
• asaside-effectgivesusextraredundancy• Developmentloadistoosmalltocatchmanyissues• Recoveriesdisabledglobally onthisclusterbutmonitoringworksthesame
• Complianceregulationsmayrequiresegregationofdifferentnetworks
29
OrchestratorHA
Solution• MovefromusingorchestratorbinarytouseclusterAPIinterface
• Recentlymigratedtouseneworchestrator-clientcommandwhichsolvesthesameproblemandwasneededfororchestrator/raftaccess
• Simplifiesconfiguration• Allowseasyaccesstomorethenoneorchestratorcluster• Orchestratorupgradeswithdb backendchangesareeasier
30
OrchestratorAPI
EnhancementstoAPIcalls• Bulkretrievalofinstanceinformationandpromotionrules• Asynchronousdiscoverycall(e.g.bootstrapnewcluster)• Moremonitoringinformationavailable
• Discoverytimingmetrics• Discoveryqueuemetrics• Backendwritemetrics
31
SpecialCases
• TestingMySQL8.0orMariaDB 10.3?• “Let’snotpromotetothisbox”• Sameapplieswhiletestingnewminorversionsofcourse
• Sometopologieshaveslaveswithaggregatedata• Donot treatthemasanormalbox– shouldnot becandidatemasters
• OrchestratorcannothandleGRormulti-sourcereplicationyet• Besttoavoid theseboxes(forautomaticfailover)untilwehavesolutions• Patcheswelcometosolvesuchmissingfunctionality
32
SpecialCases
HandlingTLSconnections• OrchestratorcouldhandleusingTLSornotusingitbut…• SomeserversneedtobeaccessedbyTLS,othersdon’t(ODBCaccessormoresecuritysensitive systems)
• Orchestratorcouldnothandlethis• Codeaddedtorecognise errorandautomaticallyswitchtoTLS:
• Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON
• GlobalOFF button– givesyoupeaceofmind
33
ProvideWiderUserAccess
• Orchestratorfanclub• Differentgroupsofuserslikeorchestrator• DBAs,Developers,Sysadmins,Auditors,Managers
• Usenginx (orsimilar)• Providesauthentication• ProvidesTLS• Thecombinationcanbeusedwithunix groupstoallowuser oradmin accesstoorchestrator
• Combinedwithaloadbalancerprovideseasyaccessforusersandalsoforapplications(usingapi calls)
34
Monitoring
Somethingstomonitor• Orchestratorprocess(andnginx)• Orchestratorclusterendpoint• SuccessfulorfailedDiscoveriesperminute• Discoveryqueuesizes• Discoverytimings
• aggregatedatagivesmean,medianandpercentiles• DiscoveriesexceedingInstancePollSeconds• Whenchangingactiveorchestratornodethesevaluesmay change
35
Booking.com contributions
Commitstopublicorchestratorrepo• Simon:170• Dmitry:40• Mauro:15• Daniël:8• Shlomi:many(whileworkingatbooking)
36
Usingorchestratoratsmallerscale
37
Usingorchestratoratsmallerscale
Notmentionedherebut• ConsideruseofSqlite – goodstartingpoint– singlebinary• ConsideruseofSqlite/raft
• providesHA• allnodesmonitorallMySQLservers
• Onlydifferenceisthedb backend• Notsurewherescalinglimits
38
Configurationsettings
Settingstobeconsidered,brokendownbyfunction
39
Configurationsettings
• MySQLbackend• "MySQLOrchestratorCredentialsConfigFile":"/path/.my-orchestratordb.cnf"• "MySQLOrchestratorDatabase":"orchestrator”• "MySQLOrchestratorHost":"orchestratordb.example.com"• "MySQLOrchestratorPort":3306• "MySQLOrchestratorMaxPoolConnections":100• "MySQLConnectTimeoutSeconds":1
• Sqlite backend• "BackendDB":"sqlite”• "SQLite3DataFile":"/var/lib/orchestrator/orchestrator.db"
40
Configurationsettings
• Psuedo-GTIDSettings(ifusingpseudo-gtid)• PseudoGTIDPattern• PseudoGTIDMonotonicHint• DetectPseudoGTIDQuery
41
Configurationsettings
• Clusterandhostsettings• Querymetadatadb (populatedexternally)todetectclusters• DetectClusterAliasQuery• DetectClusterDomainQuery
42
Configurationsettings
• Recoverysettings• Regexp filters– verysitedependent• RecoverMasterClusterFilters – white-listmastersbyclustername• RecoverIntermediateMasterClusterFilters• PromotionIgnoreHostnameFilters – ignoreserversfrombeingpromoted*• RecoveryIgnoreHostnameFilters – ignorespecialserversfromrecovery
43*Doesnotscalewell
Configurationsettings
• Failoversettings• OnFailureDetectionProcesses – whattodowhenafailureisdetected• PreFailoverProcesses – whattodopriortostartingrecovery• PostFailoverProcesses – whattodoaftercompletingrecovery• PostUnsuccessfulFailoverProcesses – whattodoifrecoveryfails• PostMasterFailoverProcesses – whattodoafterIMrecovery• PostIntermediateMasterFailoverProcesses – whattodoafterMasterrecovery
44
Configurationsettings
• Authenticationsettings(e.g.ifusingnginx withLDAP)• "AuthenticationMethod":"proxy",• "HTTPAuthUser":”user1",• "HTTPAuthPassword":”pass1",• "AuthUserHeader":”SomeHeader",• “PowerAuthUsers":["api-user1","api-user2",”realuser1"]• PowerAuthGroups":[”special_sysadmins”,“dbas”],
45*Doesnotscalewell
Configurationsettings
• Environmentsettings(e.g.shorten/simplifyhostnames)• “DataCenterPattern”• “PhysicalEnvironmentPattern”:• “RemoveTextFromHostnameDisplay”:“:.example.com:3306”
46
WayForward
47
WayForward
• Improvementsneededtotackleproblemsatbothendsofthescale• Smallerinstallations– forgettingonboard• Largerinstallations– toallowforfurtherscaling
48
WayForward
• Simplifyconfigurationandentrytoorchestrator• Shlomi isdoingaverygoodjobwithsqlite andraftsetups• Configurationcouldbesimplerandmoreautomaticformostpeople• Needtostandardise orchestratorsetupsmore?
• ExtendfunctionalitytocovermoreoftheMySQLeco-system• AWSandothercloudsystems• GroupReplicationorGalera• Multi-source
49
WayForward
• Distributionofdiscoveriesamongstallorchestratornodes• Orchestrator/raft:allnodesmonitorallMySQLservers
• Raftusagerecommendshavingseveralnodes• Orchestrator/MySQL:onenodemonitorsallMySQLservers• Better:distributemonitoringamongstavailablenodes
• Avoidsunnecessaryloadonmonitoredservers• reducesworkonbusyorchestratorapps• Usefulforsmallandlargeinstallations
• efficientbalancingisharder
50
WayForward
• Reducerecoverytime• Speedingupdetectiontorecoverytime wouldbegoodasreducesdowntime• Shouldbepossibletoreacttofailureevent(knowingstateofotherservers)immediately
• statecurrentlystoredinbackenddb• analysisanddetectionphasehappensindependentlyofserverpolling
• Withreduceddefaultpolltimeof5secondsrecoveryislikelytobetriggeredwithin10seconds
• notcriticalformostpeople?
51
WayForward
• Furtherworkneededtoscalemore• bottlenecksstillexist• Largerinstallationskeepgrowing
• Improvemonitoring• ExternalAPIcalls• Addinternalmetrics
52
Conclusion
53
Doesitwork?
Icheckedforfailuresoverarecentperiod• 6masterfailures• About40intermediatemasterfailures• No-onecalledup• Noharmwasdone
54
Questions?
Thanks
Simon J [email protected]